2024-09-14T22:39:07.928 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-14T22:39:07.949 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T22:39:08.053 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733 branch: squid description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: ceph-qa@ceph.com first_in_suite: false job_id: '7904733' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 4271 sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 sleep_before_teardown: 0 subset: 97/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPg450sCgqSZFEMi05UZtTnatCArbuCBeU52Co/YZUkC5+nBt0mPDFFs+mOtIM2CF14NE+PwBDTR8HrPFF6sMvg= smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOphTnuc232wFEHlDEc7DtT8nVVpon/sJ/e1bz9jx+3kNyW0F3b1ZugF+/ty2d3A/yCN1Y97GFGyi2SJGs5K0AM= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-14_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-14T22:39:08.054 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa; will attempt to use it 2024-09-14T22:39:08.054 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa/tasks 2024-09-14T22:39:08.054 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-14T22:39:08.056 INFO:teuthology.task.internal:Checking packages... 2024-09-14T22:39:08.078 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ecab7be58963a5bd03910cd73eef66405d7fdd50' 2024-09-14T22:39:08.078 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T22:39:08.078 INFO:teuthology.packaging:ref: None 2024-09-14T22:39:08.078 INFO:teuthology.packaging:tag: None 2024-09-14T22:39:08.078 INFO:teuthology.packaging:branch: squid 2024-09-14T22:39:08.078 INFO:teuthology.packaging:sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-14T22:39:08.079 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-14T22:39:08.296 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-324.gecab7be5 2024-09-14T22:39:08.298 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-14T22:39:08.305 INFO:teuthology.task.internal:no buildpackages task found 2024-09-14T22:39:08.305 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-14T22:39:08.322 INFO:teuthology.task.internal:Saving configuration 2024-09-14T22:39:08.338 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-14T22:39:08.345 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-14T22:39:08.370 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 22:34:04.012796', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPg450sCgqSZFEMi05UZtTnatCArbuCBeU52Co/YZUkC5+nBt0mPDFFs+mOtIM2CF14NE+PwBDTR8HrPFF6sMvg='} 2024-09-14T22:39:08.390 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 22:34:04.015206', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOphTnuc232wFEHlDEc7DtT8nVVpon/sJ/e1bz9jx+3kNyW0F3b1ZugF+/ty2d3A/yCN1Y97GFGyi2SJGs5K0AM='} 2024-09-14T22:39:08.390 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-14T22:39:08.397 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-14T22:39:08.397 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-14T22:39:08.397 INFO:teuthology.run_tasks:Running task console_log... 2024-09-14T22:39:08.463 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0a142cfac0>, signals=[15]) 2024-09-14T22:39:08.463 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-14T22:39:08.479 INFO:teuthology.task.internal:Opening connections... 2024-09-14T22:39:08.479 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T22:39:08.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:39:08.562 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T22:39:08.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:39:08.640 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-14T22:39:08.646 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-09-14T22:39:08.665 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-09-14T22:39:08.665 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-09-14T22:39:08.722 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-09-14T22:39:08.722 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-09-14T22:39:08.722 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-09-14T22:39:08.722 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-09-14T22:39:08.722 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-09-14T22:39:08.722 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-09-14T22:39:08.722 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-14T22:39:08.722 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-09-14T22:39:08.722 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-09-14T22:39:08.723 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-14T22:39:08.723 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-09-14T22:39:08.723 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-14T22:39:08.723 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-14T22:39:08.723 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-14T22:39:08.723 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-09-14T22:39:08.751 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-09-14T22:39:08.774 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-09-14T22:39:08.774 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-09-14T22:39:08.831 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-14T22:39:08.832 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-14T22:39:08.833 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-14T22:39:08.833 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-09-14T22:39:08.877 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-14T22:39:08.887 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-14T22:39:08.913 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-14T22:39:08.913 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-09-14T22:39:08.916 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-09-14T22:39:08.932 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-14T22:39:08.943 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-14T22:39:08.943 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-09-14T22:39:08.973 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-09-14T22:39:09.008 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-14T22:39:09.055 INFO:teuthology.run_tasks:Running task kernel... 2024-09-14T22:39:09.070 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-14T22:39:09.070 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-14T22:39:09.070 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T22:39:09.071 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T22:39:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T22:39:09.087 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-09-14T22:39:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T22:39:09.088 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-09-14T22:39:09.145 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-503.el9.x86_64 2024-09-14T22:39:09.145 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-503.el9.x86_64 2024-09-14T22:39:09.145 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-09-14T22:39:09.147 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-503.el9.x86_64 2024-09-14T22:39:09.147 INFO:teuthology.task.kernel:Running kernel on smithi089: 5.14.0-503.el9.x86_64 2024-09-14T22:39:09.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2024-09-14T22:39:11.456 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-14T22:39:12.490 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-14T22:39:13.815 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 16 MB/s | 20 MB 00:01 2024-09-14T22:39:15.616 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-09-14T22:39:17.965 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 19 MB/s | 6.5 MB 00:00 2024-09-14T22:39:19.575 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-14T22:39:20.404 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-14T22:39:20.883 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.5 MB 00:01 2024-09-14T22:39:22.625 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 19 kB 00:00 2024-09-14T22:39:23.396 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-14T22:39:25.810 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-14T22:39:27.808 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-14T22:39:27.808 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-14T22:39:27.902 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:39:27.910 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:39:27.911 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2024-09-14T22:39:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:39:27.912 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 79 M 2024-09-14T22:39:27.912 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 126 M 2024-09-14T22:39:27.912 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:39:28.494 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 43 kB/s | 23 kB 00:00 2024-09-14T22:39:29.152 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-14T22:39:29.378 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-14T22:39:30.070 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-14T22:39:30.712 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-14T22:39:30.712 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:39:30.712 INFO:teuthology.orchestra.run.smithi089.stdout:Total 28 MB/s | 79 MB 00:02 2024-09-14T22:39:31.192 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-14T22:39:31.192 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-14T22:39:31.287 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:39:31.298 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:39:31.299 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:39:31.300 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-09-14T22:39:31.300 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:39:31.300 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 79 M 2024-09-14T22:39:31.300 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 126 M 2024-09-14T22:39:31.300 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:39:31.391 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:39:31.437 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 198 kB/s | 23 kB 00:00 2024-09-14T22:39:31.732 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:39:31.732 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:39:32.104 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-09-14T22:39:32.612 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-14T22:39:32.690 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:39:32.690 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:39:33.096 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 17 MB/s | 29 MB 00:01 2024-09-14T22:39:33.096 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:39:33.096 INFO:teuthology.orchestra.run.smithi052.stdout:Total 44 MB/s | 79 MB 00:01 2024-09-14T22:39:33.798 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:39:34.186 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:39:34.186 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:39:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:39:35.167 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:39:35.168 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:39:35.848 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-14T22:39:35.954 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T22:39:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:39:38.398 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-14T22:39:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T22:39:38.754 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T22:39:38.891 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T22:39:41.272 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T22:39:41.401 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T22:39:43.694 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T22:39:43.758 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:39:46.533 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T22:39:46.577 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:39:48.598 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:39:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:40:27.066 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:40:27.106 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:40:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:40:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-14T22:40:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T22:40:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T22:40:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:40:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:40:28.329 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:40:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-14T22:40:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-14T22:40:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-14T22:40:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-14T22:40:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:40:28.329 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:40:28.677 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T22:40:28.771 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T22:40:28.771 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T22:40:28.771 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-14T22:40:28.771 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T22:40:28.772 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T22:40:28.772 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-14T22:40:28.772 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi089.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T22:40:28.772 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T22:40:28.830 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T22:40:28.831 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T22:40:28.831 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -qi grub2-tools 2024-09-14T22:40:28.863 INFO:teuthology.orchestra.run.smithi089.stdout:Name : grub2-tools 2024-09-14T22:40:28.863 INFO:teuthology.orchestra.run.smithi089.stdout:Epoch : 1 2024-09-14T22:40:28.863 INFO:teuthology.orchestra.run.smithi089.stdout:Version : 2.06 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Release : 82.el9 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Architecture: x86_64 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Group : Unspecified 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Size : 8274670 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:License : GPLv3+ 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Packager : builder@centos.org 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:Vendor : CentOS 2024-09-14T22:40:28.864 INFO:teuthology.orchestra.run.smithi089.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-14T22:40:28.865 INFO:teuthology.orchestra.run.smithi089.stdout:Summary : Support tools for GRUB. 2024-09-14T22:40:28.865 INFO:teuthology.orchestra.run.smithi089.stdout:Description : 2024-09-14T22:40:28.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:40:28.865 INFO:teuthology.orchestra.run.smithi089.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-14T22:40:28.865 INFO:teuthology.orchestra.run.smithi089.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-14T22:40:28.865 INFO:teuthology.orchestra.run.smithi089.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-14T22:40:28.865 INFO:teuthology.orchestra.run.smithi089.stdout:hardware devices. 2024-09-14T22:40:28.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:40:28.865 INFO:teuthology.orchestra.run.smithi089.stdout:This subpackage provides tools for support of all platforms. 2024-09-14T22:40:28.866 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-14T22:40:28.866 INFO:teuthology.task.kernel:Updating grub on smithi089 to boot 5.14.0-505.el9.x86_64 2024-09-14T22:40:28.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-14T22:40:29.491 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-09-14T22:40:30.841 INFO:teuthology.orchestra.run.smithi089.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-14T22:40:30.867 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-09-14T22:40:30.869 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-09-14T22:40:30.885 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.bT2HiJWWDd 2024-09-14T22:40:30.885 DEBUG:teuthology.orchestra.run.smithi089:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bT2HiJWWDd 2024-09-14T22:40:30.951 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.bT2HiJWWDd 2024-09-14T22:40:31.096 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.bT2HiJWWDd is 6KB 2024-09-14T22:40:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:40:31.144 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.bT2HiJWWDd 2024-09-14T22:40:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:40:31.159 DEBUG:teuthology.orchestra.run.smithi089:> sudo /bin/ls /boot/loader/entries || true 2024-09-14T22:40:31.222 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-14T22:40:31.222 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-14T22:40:31.222 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-14T22:40:31.222 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-14T22:40:31.223 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-14T22:40:31.376 DEBUG:teuthology.orchestra.run.smithi089:> sudo shutdown -r now 2024-09-14T22:40:31.530 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:40:31.530 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-14T22:40:31.530 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T22:40:31.530 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T22:40:32.337 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T22:40:32.338 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:40:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:40:32.338 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-14T22:40:32.338 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-14T22:40:32.338 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-14T22:40:32.338 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-14T22:40:32.338 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:40:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:40:32.682 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T22:40:32.767 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T22:40:32.767 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T22:40:32.768 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-14T22:40:32.768 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T22:40:32.768 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T22:40:32.768 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-14T22:40:32.768 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T22:40:32.768 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T22:40:32.827 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T22:40:32.828 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T22:40:32.828 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-09-14T22:40:32.859 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-09-14T22:40:32.859 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-09-14T22:40:32.859 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-09-14T22:40:32.859 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 82.el9 2024-09-14T22:40:32.859 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-09-14T22:40:32.859 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-14T22:40:32.859 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8274670 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-14T22:40:32.860 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-09-14T22:40:32.861 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-09-14T22:40:32.861 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:40:32.861 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-14T22:40:32.861 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-14T22:40:32.861 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-14T22:40:32.861 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-09-14T22:40:32.861 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:40:32.861 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-09-14T22:40:32.862 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-14T22:40:32.862 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-505.el9.x86_64 2024-09-14T22:40:32.862 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-14T22:40:33.502 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-09-14T22:40:34.853 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-14T22:40:34.889 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-09-14T22:40:34.890 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-09-14T22:40:34.932 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.wGHoox0cI8 2024-09-14T22:40:34.933 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wGHoox0cI8 2024-09-14T22:40:34.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.wGHoox0cI8 2024-09-14T22:40:35.140 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.wGHoox0cI8 is 6KB 2024-09-14T22:40:35.161 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.wGHoox0cI8 2024-09-14T22:40:35.175 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-09-14T22:40:35.237 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-14T22:40:35.238 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-14T22:40:35.238 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-14T22:40:35.238 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-14T22:40:35.239 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-14T22:40:35.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-09-14T22:41:01.404 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-14T22:41:01.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-14T22:41:01.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:41:05.401 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-14T22:41:05.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-14T22:41:05.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:41:19.933 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-09-14T22:41:28.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-14T22:41:28.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:41:31.997 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-09-14T22:41:44.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-14T22:41:44.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:42:02.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-09-14T22:42:05.405 DEBUG:teuthology.orchestra.remote:timed out 2024-09-14T22:42:14.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-14T22:42:14.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:42:14.792 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-09-14T22:42:15.283 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-14T22:42:15.283 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-14T22:42:15.283 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-09-14T22:42:15.299 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-505.el9.x86_64 2024-09-14T22:42:15.299 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-14T22:42:15.299 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T22:42:15.299 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-14T22:42:16.300 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-14T22:42:16.300 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T22:42:16.378 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-09-14T22:42:16.421 DEBUG:teuthology.parallel:result is None 2024-09-14T22:42:17.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-14T22:42:17.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:42:18.106 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-09-14T22:42:18.615 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-14T22:42:18.615 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-14T22:42:18.615 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-09-14T22:42:18.654 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-505.el9.x86_64 2024-09-14T22:42:18.654 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-14T22:42:18.655 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T22:42:18.655 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-14T22:42:19.655 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-14T22:42:19.656 DEBUG:teuthology.orchestra.run.smithi089:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T22:42:19.711 INFO:teuthology.orchestra.run.smithi089.stdout:ttyS1 2024-09-14T22:42:19.735 DEBUG:teuthology.parallel:result is None 2024-09-14T22:42:19.735 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-14T22:42:19.748 INFO:teuthology.task.internal:Creating test directory... 2024-09-14T22:42:19.748 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T22:42:19.752 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T22:42:19.769 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-14T22:42:19.775 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-14T22:42:19.810 INFO:teuthology.task.internal:Creating archive directory... 2024-09-14T22:42:19.810 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T22:42:19.812 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T22:42:19.848 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-14T22:42:19.855 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-14T22:42:19.855 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T22:42:19.873 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T22:42:19.897 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T22:42:19.905 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T22:42:19.908 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T22:42:19.915 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T22:42:19.916 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-14T22:42:19.923 INFO:teuthology.task.internal:Configuring sudo... 2024-09-14T22:42:19.924 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T22:42:19.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T22:42:19.979 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-14T22:42:19.988 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-14T22:42:19.988 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T22:42:20.026 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T22:42:20.041 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T22:42:20.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T22:42:20.179 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T22:42:20.265 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T22:42:20.309 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:42:20.309 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T22:42:20.378 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T22:42:20.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T22:42:20.466 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T22:42:20.549 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T22:42:20.585 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:42:20.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T22:42:20.648 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-09-14T22:42:20.651 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-09-14T22:42:20.701 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T22:42:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T22:42:21.208 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-14T22:42:21.216 INFO:teuthology.task.internal:Starting timer... 2024-09-14T22:42:21.216 INFO:teuthology.run_tasks:Running task pcp... 2024-09-14T22:42:21.279 INFO:teuthology.run_tasks:Running task selinux... 2024-09-14T22:42:21.303 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-14T22:42:21.303 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-09-14T22:42:21.388 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-09-14T22:42:21.390 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-09-14T22:42:21.429 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-09-14T22:42:21.431 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-14T22:42:21.431 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-09-14T22:42:21.480 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-09-14T22:42:21.481 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-09-14T22:42:21.517 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-09-14T22:42:21.517 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi089.front.sepia.ceph.com': 'permissive'} 2024-09-14T22:42:21.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-14T22:42:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T22:42:21.543 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-14T22:42:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T22:42:21.582 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-14T22:42:21.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-09-14T22:42:21.891 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-09-14T22:42:21.923 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-14T22:42:21.932 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-14T22:42:21.936 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-14T22:42:22.046 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-14T22:42:22.057 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-14T22:42:22.058 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi089.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-14T22:46:05.440 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi089.front.sepia.ceph.com')] 2024-09-14T22:46:05.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-14T22:46:05.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:46:05.521 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-09-14T22:46:05.609 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-14T22:46:05.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-14T22:46:05.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T22:46:05.685 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-09-14T22:46:05.768 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-14T22:46:05.769 INFO:teuthology.run_tasks:Running task clock... 2024-09-14T22:46:05.780 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-14T22:46:05.780 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T22:46:05.781 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T22:46:05.783 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T22:46:05.783 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T22:46:05.812 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-14T22:46:05.828 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-14T22:46:05.839 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-14T22:46:05.853 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-14T22:46:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-09-14T22:46:05.866 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-09-14T22:46:05.878 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-09-14T22:46:05.882 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-14T22:46:05.889 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-09-14T22:46:05.896 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-14T22:46:05.903 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-14T22:46:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-14T22:46:05.951 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-09-14T22:46:05.954 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T22:46:05.955 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-09-14T22:46:05.955 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T22:46:05.955 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T22:46:05.955 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T22:46:05.955 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T22:46:05.971 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-09-14T22:46:05.974 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T22:46:05.974 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-09-14T22:46:05.975 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T22:46:05.975 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T22:46:05.975 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T22:46:05.975 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T22:46:05.976 INFO:teuthology.run_tasks:Running task install... 2024-09-14T22:46:05.984 DEBUG:teuthology.task.install:project ceph 2024-09-14T22:46:05.984 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}} 2024-09-14T22:46:05.984 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'} 2024-09-14T22:46:05.984 INFO:teuthology.task.install:Using flavor: default 2024-09-14T22:46:05.990 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-14T22:46:05.990 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-14T22:46:05.990 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'tag': None, 'wait_for_package': False} 2024-09-14T22:46:05.991 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-14T22:46:05.993 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'tag': None, 'wait_for_package': False} 2024-09-14T22:46:05.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-14T22:46:06.135 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/ 2024-09-14T22:46:06.135 INFO:teuthology.task.install.rpm:Package version is 19.1.1-324.gecab7be5 2024-09-14T22:46:06.140 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/ 2024-09-14T22:46:06.140 INFO:teuthology.task.install.rpm:Package version is 19.1.1-324.gecab7be5 2024-09-14T22:46:06.248 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-14T22:46:06.248 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:46:06.248 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-14T22:46:06.253 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-14T22:46:06.253 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:46:06.253 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-14T22:46:06.276 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-14T22:46:06.276 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ecab7be58963a5bd03910cd73eef66405d7fdd50/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-14T22:46:06.281 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-14T22:46:06.281 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ecab7be58963a5bd03910cd73eef66405d7fdd50/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-14T22:46:06.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-14T22:46:06.347 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-14T22:46:06.419 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-14T22:46:06.422 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-14T22:46:06.484 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-09-14T22:46:06.486 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-09-14T22:46:06.486 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-09-14T22:46:06.488 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-09-14T22:46:06.730 INFO:teuthology.orchestra.run.smithi052.stdout:45 files removed 2024-09-14T22:46:06.735 INFO:teuthology.orchestra.run.smithi089.stdout:45 files removed 2024-09-14T22:46:06.757 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2024-09-14T22:46:06.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw 2024-09-14T22:46:07.319 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 305 kB/s | 84 kB 00:00 2024-09-14T22:46:07.333 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 301 kB/s | 84 kB 00:00 2024-09-14T22:46:07.603 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-14T22:46:07.625 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-14T22:46:07.879 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-14T22:46:07.901 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-14T22:46:08.392 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-14T22:46:10.304 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.2 MB 00:02 2024-09-14T22:46:12.339 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-14T22:46:12.918 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 20 MB 00:03 2024-09-14T22:46:16.506 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-14T22:46:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.5 MB 00:00 2024-09-14T22:46:17.899 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 103 kB/s | 19 kB 00:00 2024-09-14T22:46:18.653 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-14T22:46:20.153 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-14T22:46:20.957 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-14T22:46:24.143 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-09-14T22:46:26.511 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:46:26.522 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:46:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:46:26.522 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:46:26.522 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:46:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:19.1.1-324.gecab7be5.el9 ceph 11 M 2024-09-14T22:46:26.522 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-09-14T22:46:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.4 M 2024-09-14T22:46:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.2 M 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:19.1.1-324.gecab7be5.el9 ceph 5.4 M 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:19.1.1-324.gecab7be5.el9 ceph 21 M 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:19.1.1-324.gecab7be5.el9 ceph 25 k 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 752 k 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-14T22:46:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 507 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 5.3 M 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 45 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:19.1.1-324.gecab7be5.el9 ceph 136 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:19.1.1-324.gecab7be5.el9 ceph 162 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:19.1.1-324.gecab7be5.el9 ceph 321 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 301 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:19.1.1-324.gecab7be5.el9 ceph 99 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-14T22:46:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout:Installing weak dependencies: 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout:Install 30 Packages 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 2 Packages 2024-09-14T22:46:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:46:26.526 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 60 M 2024-09-14T22:46:26.526 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:46:26.696 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-14T22:46:27.302 INFO:teuthology.orchestra.run.smithi089.stdout:(1/32): ceph-base-19.1.1-324.gecab7be5.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-14T22:46:27.369 INFO:teuthology.orchestra.run.smithi089.stdout:(2/32): ceph-selinux-19.1.1-324.gecab7be5.el9.x 378 kB/s | 25 kB 00:00 2024-09-14T22:46:27.462 INFO:teuthology.orchestra.run.smithi089.stdout:(3/32): libcephfs2-19.1.1-324.gecab7be5.el9.x86 8.0 MB/s | 752 kB 00:00 2024-09-14T22:46:27.545 INFO:teuthology.orchestra.run.smithi089.stdout:(4/32): libradosstriper1-19.1.1-324.gecab7be5.e 6.0 MB/s | 507 kB 00:00 2024-09-14T22:46:27.679 INFO:teuthology.orchestra.run.smithi089.stdout:(5/32): ceph-radosgw-19.1.1-324.gecab7be5.el9.x 12 MB/s | 11 MB 00:00 2024-09-14T22:46:27.746 INFO:teuthology.orchestra.run.smithi089.stdout:(6/32): python3-ceph-argparse-19.1.1-324.gecab7 678 kB/s | 45 kB 00:00 2024-09-14T22:46:27.829 INFO:teuthology.orchestra.run.smithi089.stdout:(7/32): python3-ceph-common-19.1.1-324.gecab7be 1.6 MB/s | 136 kB 00:00 2024-09-14T22:46:27.905 INFO:teuthology.orchestra.run.smithi089.stdout:(8/32): librgw2-19.1.1-324.gecab7be5.el9.x86_64 15 MB/s | 5.3 MB 00:00 2024-09-14T22:46:27.930 INFO:teuthology.orchestra.run.smithi089.stdout:(9/32): python3-cephfs-19.1.1-324.gecab7be5.el9 1.6 MB/s | 162 kB 00:00 2024-09-14T22:46:27.955 INFO:teuthology.orchestra.run.smithi089.stdout:(10/32): python3-rados-19.1.1-324.gecab7be5.el9 6.2 MB/s | 321 kB 00:00 2024-09-14T22:46:27.981 INFO:teuthology.orchestra.run.smithi089.stdout:(11/32): python3-rbd-19.1.1-324.gecab7be5.el9.x 5.8 MB/s | 301 kB 00:00 2024-09-14T22:46:28.006 INFO:teuthology.orchestra.run.smithi089.stdout:(12/32): python3-rgw-19.1.1-324.gecab7be5.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-14T22:46:28.231 INFO:teuthology.orchestra.run.smithi089.stdout:(13/32): ceph-common-19.1.1-324.gecab7be5.el9.x 15 MB/s | 21 MB 00:01 2024-09-14T22:46:28.315 INFO:teuthology.orchestra.run.smithi089.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 320 kB/s | 107 kB 00:00 2024-09-14T22:46:28.340 INFO:teuthology.orchestra.run.smithi089.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-09-14T22:46:28.374 INFO:teuthology.orchestra.run.smithi089.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 435 kB/s | 160 kB 00:00 2024-09-14T22:46:28.399 INFO:teuthology.orchestra.run.smithi089.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-14T22:46:28.433 INFO:teuthology.orchestra.run.smithi089.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-14T22:46:28.450 INFO:teuthology.orchestra.run.smithi089.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 559 kB/s | 42 kB 00:00 2024-09-14T22:46:28.467 INFO:teuthology.orchestra.run.smithi089.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 36 kB 00:00 2024-09-14T22:46:28.484 INFO:teuthology.orchestra.run.smithi089.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-14T22:46:28.518 INFO:teuthology.orchestra.run.smithi089.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-14T22:46:28.568 INFO:teuthology.orchestra.run.smithi089.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-14T22:46:28.593 INFO:teuthology.orchestra.run.smithi089.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-14T22:46:28.618 INFO:teuthology.orchestra.run.smithi089.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 300 kB/s | 26 kB 00:00 2024-09-14T22:46:28.644 INFO:teuthology.orchestra.run.smithi089.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-14T22:46:28.677 INFO:teuthology.orchestra.run.smithi089.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-14T22:46:28.719 INFO:teuthology.orchestra.run.smithi089.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-14T22:46:28.778 INFO:teuthology.orchestra.run.smithi089.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-14T22:46:28.853 INFO:teuthology.orchestra.run.smithi089.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-14T22:46:28.987 INFO:teuthology.orchestra.run.smithi089.stdout:(31/32): librados2-19.1.1-324.gecab7be5.el9.x86 13 MB/s | 3.4 MB 00:00 2024-09-14T22:46:29.054 INFO:teuthology.orchestra.run.smithi089.stdout:(32/32): librbd1-19.1.1-324.gecab7be5.el9.x86_6 11 MB/s | 3.2 MB 00:00 2024-09-14T22:46:29.059 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:46:29.059 INFO:teuthology.orchestra.run.smithi089.stdout:Total 24 MB/s | 60 MB 00:02 2024-09-14T22:46:29.163 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:46:29.199 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:46:29.199 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:46:29.227 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:19.1.1-324.gecab7be5.el9 ceph 11 M 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.4 M 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.2 M 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-14T22:46:29.231 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:19.1.1-324.gecab7be5.el9 ceph 5.4 M 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:19.1.1-324.gecab7be5.el9 ceph 21 M 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:19.1.1-324.gecab7be5.el9 ceph 25 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 752 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 507 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:19.1.1-324.gecab7be5.el9 ceph 5.3 M 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-14T22:46:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 45 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:19.1.1-324.gecab7be5.el9 ceph 136 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:19.1.1-324.gecab7be5.el9 ceph 162 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:19.1.1-324.gecab7be5.el9 ceph 321 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 301 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:19.1.1-324.gecab7be5.el9 ceph 99 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-14T22:46:29.233 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2024-09-14T22:46:29.234 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-14T22:46:29.234 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:46:29.234 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:46:29.234 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:46:29.234 INFO:teuthology.orchestra.run.smithi052.stdout:Install 30 Packages 2024-09-14T22:46:29.234 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2024-09-14T22:46:29.234 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:46:29.234 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 60 M 2024-09-14T22:46:29.234 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:46:29.876 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:46:29.876 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:46:30.164 INFO:teuthology.orchestra.run.smithi052.stdout:(1/32): ceph-base-19.1.1-324.gecab7be5.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-14T22:46:30.223 INFO:teuthology.orchestra.run.smithi052.stdout:(2/32): ceph-selinux-19.1.1-324.gecab7be5.el9.x 432 kB/s | 25 kB 00:00 2024-09-14T22:46:30.315 INFO:teuthology.orchestra.run.smithi052.stdout:(3/32): libcephfs2-19.1.1-324.gecab7be5.el9.x86 8.0 MB/s | 752 kB 00:00 2024-09-14T22:46:30.415 INFO:teuthology.orchestra.run.smithi052.stdout:(4/32): libradosstriper1-19.1.1-324.gecab7be5.e 5.0 MB/s | 507 kB 00:00 2024-09-14T22:46:30.574 INFO:teuthology.orchestra.run.smithi052.stdout:(5/32): ceph-radosgw-19.1.1-324.gecab7be5.el9.x 12 MB/s | 11 MB 00:00 2024-09-14T22:46:30.624 INFO:teuthology.orchestra.run.smithi052.stdout:(6/32): python3-ceph-argparse-19.1.1-324.gecab7 907 kB/s | 45 kB 00:00 2024-09-14T22:46:30.824 INFO:teuthology.orchestra.run.smithi052.stdout:(7/32): ceph-common-19.1.1-324.gecab7be5.el9.x8 19 MB/s | 21 MB 00:01 2024-09-14T22:46:30.851 INFO:teuthology.orchestra.run.smithi052.stdout:(8/32): python3-ceph-common-19.1.1-324.gecab7be 605 kB/s | 136 kB 00:00 2024-09-14T22:46:30.875 INFO:teuthology.orchestra.run.smithi052.stdout:(9/32): python3-cephfs-19.1.1-324.gecab7be5.el9 3.2 MB/s | 162 kB 00:00 2024-09-14T22:46:30.900 INFO:teuthology.orchestra.run.smithi052.stdout:(10/32): python3-rados-19.1.1-324.gecab7be5.el9 6.3 MB/s | 321 kB 00:00 2024-09-14T22:46:30.925 INFO:teuthology.orchestra.run.smithi052.stdout:(11/32): python3-rbd-19.1.1-324.gecab7be5.el9.x 5.9 MB/s | 301 kB 00:00 2024-09-14T22:46:31.000 INFO:teuthology.orchestra.run.smithi052.stdout:(12/32): librgw2-19.1.1-324.gecab7be5.el9.x86_6 9.1 MB/s | 5.3 MB 00:00 2024-09-14T22:46:31.034 INFO:teuthology.orchestra.run.smithi052.stdout:(13/32): python3-rgw-19.1.1-324.gecab7be5.el9.x 740 kB/s | 99 kB 00:00 2024-09-14T22:46:31.118 INFO:teuthology.orchestra.run.smithi052.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 556 kB/s | 107 kB 00:00 2024-09-14T22:46:31.143 INFO:teuthology.orchestra.run.smithi052.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-09-14T22:46:31.168 INFO:teuthology.orchestra.run.smithi052.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-14T22:46:31.210 INFO:teuthology.orchestra.run.smithi052.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-09-14T22:46:31.235 INFO:teuthology.orchestra.run.smithi052.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 632 kB/s | 42 kB 00:00 2024-09-14T22:46:31.261 INFO:teuthology.orchestra.run.smithi052.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-14T22:46:31.286 INFO:teuthology.orchestra.run.smithi052.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 716 kB/s | 36 kB 00:00 2024-09-14T22:46:31.311 INFO:teuthology.orchestra.run.smithi052.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-14T22:46:31.336 INFO:teuthology.orchestra.run.smithi052.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-14T22:46:31.395 INFO:teuthology.orchestra.run.smithi052.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-14T22:46:31.420 INFO:teuthology.orchestra.run.smithi052.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-09-14T22:46:31.445 INFO:teuthology.orchestra.run.smithi052.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 972 kB/s | 49 kB 00:00 2024-09-14T22:46:31.471 INFO:teuthology.orchestra.run.smithi052.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-14T22:46:31.496 INFO:teuthology.orchestra.run.smithi052.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-14T22:46:31.528 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:46:31.530 INFO:teuthology.orchestra.run.smithi052.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-09-14T22:46:31.611 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-14T22:46:31.642 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-14T22:46:31.663 INFO:teuthology.orchestra.run.smithi052.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-09-14T22:46:31.676 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-14T22:46:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-14T22:46:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-14T22:46:31.747 INFO:teuthology.orchestra.run.smithi052.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-14T22:46:31.814 INFO:teuthology.orchestra.run.smithi052.stdout:(31/32): librados2-19.1.1-324.gecab7be5.el9.x86 12 MB/s | 3.4 MB 00:00 2024-09-14T22:46:31.914 INFO:teuthology.orchestra.run.smithi052.stdout:(32/32): librbd1-19.1.1-324.gecab7be5.el9.x86_6 13 MB/s | 3.2 MB 00:00 2024-09-14T22:46:31.918 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:46:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-14T22:46:31.919 INFO:teuthology.orchestra.run.smithi052.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-14T22:46:31.981 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-14T22:46:32.025 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:46:32.065 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:46:32.065 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:46:32.397 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-14T22:46:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-14T22:46:32.580 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-14T22:46:32.669 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 8/34 2024-09-14T22:46:32.705 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 9/34 2024-09-14T22:46:32.758 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:46:32.758 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:46:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 9/34 2024-09-14T22:46:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-14T22:46:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x 11/34 2024-09-14T22:46:32.920 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 12/34 2024-09-14T22:46:32.941 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-14T22:46:33.006 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-14T22:46:33.040 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-14T22:46:33.157 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-14T22:46:33.190 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-14T22:46:33.238 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-14T22:46:33.273 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-14T22:46:33.302 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-14T22:46:33.336 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-14T22:46:33.647 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-14T22:46:33.689 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 22/34 2024-09-14T22:46:33.780 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 22/34 2024-09-14T22:46:33.839 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 23/34 2024-09-14T22:46:33.880 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-14T22:46:34.277 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:46:34.357 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-14T22:46:34.359 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-14T22:46:34.384 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 26/34 2024-09-14T22:46:34.393 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-14T22:46:34.427 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-14T22:46:34.450 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-14T22:46:34.455 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 26/34 2024-09-14T22:46:34.475 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-14T22:46:34.532 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 27/34 2024-09-14T22:46:34.570 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86 28/34 2024-09-14T22:46:34.669 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-14T22:46:34.723 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-14T22:46:35.106 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-14T22:46:35.195 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-14T22:46:35.305 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-14T22:46:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 8/34 2024-09-14T22:46:35.463 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 9/34 2024-09-14T22:46:35.527 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 9/34 2024-09-14T22:46:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-14T22:46:35.607 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x 11/34 2024-09-14T22:46:35.677 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 12/34 2024-09-14T22:46:35.698 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-14T22:46:35.763 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-14T22:46:35.789 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-14T22:46:35.900 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-14T22:46:35.931 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-14T22:46:35.982 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-14T22:46:36.013 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-14T22:46:36.042 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-14T22:46:36.075 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-14T22:46:36.408 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-14T22:46:36.454 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 22/34 2024-09-14T22:46:36.534 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 22/34 2024-09-14T22:46:36.564 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 23/34 2024-09-14T22:46:36.603 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-14T22:46:36.837 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-14T22:46:36.884 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-14T22:46:37.112 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-14T22:46:37.139 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 26/34 2024-09-14T22:46:37.211 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 26/34 2024-09-14T22:46:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 27/34 2024-09-14T22:46:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86 28/34 2024-09-14T22:46:37.462 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-14T22:46:37.504 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 30/34 2024-09-14T22:46:37.579 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 30/34 2024-09-14T22:46:37.579 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-14T22:46:37.579 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-14T22:46:37.579 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:46:37.622 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-14T22:46:39.753 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-14T22:46:39.813 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-14T22:46:40.403 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 29/34 2024-09-14T22:46:40.441 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 30/34 2024-09-14T22:46:40.524 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 30/34 2024-09-14T22:46:40.524 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-14T22:46:40.524 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-14T22:46:40.524 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:46:40.559 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-14T22:46:47.562 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-14T22:46:47.562 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-09-14T22:46:47.562 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-09-14T22:46:47.562 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-09-14T22:46:47.563 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-09-14T22:46:47.563 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-09-14T22:46:47.563 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2024-09-14T22:46:47.563 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-09-14T22:46:47.563 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:46:48.558 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 32/34 2024-09-14T22:46:48.584 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 32/34 2024-09-14T22:46:48.584 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:46:48.584 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-14T22:46:48.584 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T22:46:48.584 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T22:46:48.584 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:46:48.586 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-14T22:46:48.661 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-14T22:46:48.661 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-14T22:46:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 1/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 3/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 4/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 5/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 6/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x 8/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86 9/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 10/34 2024-09-14T22:46:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 11/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 12/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-14T22:46:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-14T22:46:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-14T22:46:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-14T22:46:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-14T22:46:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-14T22:46:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-14T22:46:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-14T22:46:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-14T22:46:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-14T22:46:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 33/34 2024-09-14T22:46:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-14T22:46:50.858 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-09-14T22:46:50.858 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-09-14T22:46:50.858 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-09-14T22:46:50.858 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-09-14T22:46:50.858 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-09-14T22:46:50.858 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2024-09-14T22:46:50.858 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-09-14T22:46:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:46:50.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-14T22:46:50.907 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:46:50.907 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-09-14T22:46:50.907 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.907 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.907 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:46:50.907 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-14T22:46:50.908 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:50.910 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-14T22:46:50.910 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-14T22:46:50.910 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-14T22:46:50.910 INFO:teuthology.orchestra.run.smithi089.stdout: re2-1:20211101-20.el9.x86_64 2024-09-14T22:46:50.910 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-14T22:46:50.910 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:46:50.910 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:46:51.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-test 2024-09-14T22:46:51.843 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:27 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:46:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 32/34 2024-09-14T22:46:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 32/34 2024-09-14T22:46:51.895 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:46:51.895 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-14T22:46:51.895 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T22:46:51.895 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T22:46:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:46:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-14T22:46:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-14T22:46:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-14T22:46:51.980 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:46:51.981 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:46:51.981 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:46:51.981 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:46:51.981 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:46:51.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:19.1.1-324.gecab7be5.el9 ceph 49 M 2024-09-14T22:46:51.981 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-14T22:46:51.981 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:19.1.1-324.gecab7be5.el9 ceph 163 k 2024-09-14T22:46:51.981 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-14T22:46:51.982 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-14T22:46:51.982 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:46:51.982 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:46:51.982 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:46:51.982 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2024-09-14T22:46:51.982 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:46:51.982 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 49 M 2024-09-14T22:46:51.983 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 209 M 2024-09-14T22:46:51.983 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:46:52.294 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): libcephsqlite-19.1.1-324.gecab7be5.el9.x 773 kB/s | 163 kB 00:00 2024-09-14T22:46:52.877 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 383 kB/s | 304 kB 00:00 2024-09-14T22:46:52.961 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 95 kB/s | 64 kB 00:00 2024-09-14T22:46:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-14T22:46:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 1/34 2024-09-14T22:46:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2/34 2024-09-14T22:46:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 3/34 2024-09-14T22:46:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 4/34 2024-09-14T22:46:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 5/34 2024-09-14T22:46:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 6/34 2024-09-14T22:46:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 7/34 2024-09-14T22:46:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x 8/34 2024-09-14T22:46:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86 9/34 2024-09-14T22:46:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 10/34 2024-09-14T22:46:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 11/34 2024-09-14T22:46:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 12/34 2024-09-14T22:46:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 13/34 2024-09-14T22:46:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-14T22:46:53.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-14T22:46:53.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-14T22:46:53.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-14T22:46:53.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-14T22:46:53.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-14T22:46:53.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-14T22:46:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-14T22:46:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-14T22:46:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-14T22:46:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-14T22:46:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-14T22:46:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-14T22:46:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-14T22:46:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-14T22:46:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-14T22:46:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-14T22:46:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:19.1.1-324.gecab7be5.el9.x86_64 31/34 2024-09-14T22:46:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-14T22:46:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 33/34 2024-09-14T22:46:53.595 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): ceph-test-19.1.1-324.gecab7be5.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-14T22:46:53.597 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:46:53.597 INFO:teuthology.orchestra.run.smithi089.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-14T22:46:53.626 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:46:53.647 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:46:53.647 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:46:54.048 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:46:54.048 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:46:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-14T22:46:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-14T22:46:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-14T22:46:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-20.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:46:54.110 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:46:54.403 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2024-09-14T22:46:54.583 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:46:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-14T22:46:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-14T22:46:54.738 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 3/4 2024-09-14T22:46:54.992 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:28 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:46:55.130 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:46:55.130 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:19.1.1-324.gecab7be5.el9 ceph 49 M 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:19.1.1-324.gecab7be5.el9 ceph 163 k 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-09-14T22:46:55.131 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:46:55.132 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 49 M 2024-09-14T22:46:55.132 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 209 M 2024-09-14T22:46:55.132 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:46:55.372 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-14T22:46:55.397 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): libcephsqlite-19.1.1-324.gecab7be5.el9.x 684 kB/s | 163 kB 00:00 2024-09-14T22:46:55.447 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 848 kB/s | 64 kB 00:00 2024-09-14T22:46:56.672 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): ceph-test-19.1.1-324.gecab7be5.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-14T22:46:56.674 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:46:56.674 INFO:teuthology.orchestra.run.smithi052.stdout:Total 32 MB/s | 49 MB 00:01 2024-09-14T22:46:56.702 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:46:56.725 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:46:56.725 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:46:57.134 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:46:57.134 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:46:57.676 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:46:57.755 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-14T22:46:57.804 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-14T22:46:57.832 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 3/4 2024-09-14T22:46:59.375 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 3/4 2024-09-14T22:46:59.455 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 4/4 2024-09-14T22:47:01.952 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 4/4 2024-09-14T22:47:01.952 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 1/4 2024-09-14T22:47:01.952 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 2/4 2024-09-14T22:47:01.952 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-14T22:47:02.275 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-14T22:47:02.275 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:02.275 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:02.275 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:02.275 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:02.275 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-14T22:47:02.275 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-14T22:47:02.275 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:02.275 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:02.582 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph 2024-09-14T22:47:02.705 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 3/4 2024-09-14T22:47:02.783 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 4/4 2024-09-14T22:47:03.152 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:39 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:03.298 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:19.1.1-324.gecab7be5.el9 ceph 6.5 k 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:19.1.1-324.gecab7be5.el9 ceph 2.4 M 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:19.1.1-324.gecab7be5.el9 ceph 1.5 M 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 249 k 2024-09-14T22:47:03.301 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:19.1.1-324.gecab7be5.el9 ceph 4.7 M 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 17 M 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-14T22:47:03.302 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-14T22:47:03.303 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout:Install 37 Packages 2024-09-14T22:47:03.304 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:03.305 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 30 M 2024-09-14T22:47:03.305 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 108 M 2024-09-14T22:47:03.305 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:04.228 INFO:teuthology.orchestra.run.smithi089.stdout:(1/37): ceph-19.1.1-324.gecab7be5.el9.x86_64.rp 51 kB/s | 6.5 kB 00:00 2024-09-14T22:47:04.428 INFO:teuthology.orchestra.run.smithi089.stdout:(2/37): ceph-mgr-19.1.1-324.gecab7be5.el9.x86_6 4.6 MB/s | 1.5 MB 00:00 2024-09-14T22:47:04.479 INFO:teuthology.orchestra.run.smithi089.stdout:(3/37): ceph-mds-19.1.1-324.gecab7be5.el9.x86_6 6.3 MB/s | 2.4 MB 00:00 2024-09-14T22:47:04.554 INFO:teuthology.orchestra.run.smithi089.stdout:(4/37): ceph-mgr-modules-core-19.1.1-324.gecab7 3.3 MB/s | 249 kB 00:00 2024-09-14T22:47:04.688 INFO:teuthology.orchestra.run.smithi089.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-14T22:47:04.763 INFO:teuthology.orchestra.run.smithi089.stdout:(6/37): ceph-mon-19.1.1-324.gecab7be5.el9.x86_6 8.8 MB/s | 4.7 MB 00:00 2024-09-14T22:47:04.813 INFO:teuthology.orchestra.run.smithi089.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 9.8 MB/s | 1.2 MB 00:00 2024-09-14T22:47:04.855 INFO:teuthology.orchestra.run.smithi089.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-14T22:47:04.881 INFO:teuthology.orchestra.run.smithi089.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 910 kB/s | 106 kB 00:00 2024-09-14T22:47:05.065 INFO:teuthology.orchestra.run.smithi089.stdout:(10/37): ceph-osd-19.1.1-324.gecab7be5.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-14T22:47:05.098 INFO:teuthology.orchestra.run.smithi089.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 515 kB/s | 125 kB 00:00 2024-09-14T22:47:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 4/4 2024-09-14T22:47:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 1/4 2024-09-14T22:47:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 2/4 2024-09-14T22:47:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-14T22:47:05.124 INFO:teuthology.orchestra.run.smithi089.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 885 kB/s | 215 kB 00:00 2024-09-14T22:47:05.207 INFO:teuthology.orchestra.run.smithi089.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 417 kB/s | 35 kB 00:00 2024-09-14T22:47:05.233 INFO:teuthology.orchestra.run.smithi089.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-14T22:47:05.268 INFO:teuthology.orchestra.run.smithi089.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-14T22:47:05.300 INFO:teuthology.orchestra.run.smithi089.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 835 kB/s | 77 kB 00:00 2024-09-14T22:47:05.325 INFO:teuthology.orchestra.run.smithi089.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 552 kB/s | 51 kB 00:00 2024-09-14T22:47:05.351 INFO:teuthology.orchestra.run.smithi089.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 505 kB/s | 42 kB 00:00 2024-09-14T22:47:05.409 INFO:teuthology.orchestra.run.smithi089.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-14T22:47:05.435 INFO:teuthology.orchestra.run.smithi089.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 520 kB/s | 43 kB 00:00 2024-09-14T22:47:05.460 INFO:teuthology.orchestra.run.smithi089.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-14T22:47:05.481 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-14T22:47:05.482 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:05.482 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:05.482 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:05.482 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:05.482 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-14T22:47:05.482 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-14T22:47:05.482 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:05.482 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:05.485 INFO:teuthology.orchestra.run.smithi089.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-14T22:47:05.519 INFO:teuthology.orchestra.run.smithi089.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 678 kB/s | 18 kB 00:00 2024-09-14T22:47:05.553 INFO:teuthology.orchestra.run.smithi089.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-09-14T22:47:05.578 INFO:teuthology.orchestra.run.smithi089.stdout:(25/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-09-14T22:47:05.603 INFO:teuthology.orchestra.run.smithi089.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 282 kB/s | 23 kB 00:00 2024-09-14T22:47:05.629 INFO:teuthology.orchestra.run.smithi089.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-09-14T22:47:05.656 INFO:teuthology.orchestra.run.smithi089.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-14T22:47:05.679 INFO:teuthology.orchestra.run.smithi089.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-09-14T22:47:05.704 INFO:teuthology.orchestra.run.smithi089.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-14T22:47:05.730 INFO:teuthology.orchestra.run.smithi089.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-14T22:47:05.755 INFO:teuthology.orchestra.run.smithi089.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-14T22:47:05.780 INFO:teuthology.orchestra.run.smithi089.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-14T22:47:05.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2024-09-14T22:47:05.806 INFO:teuthology.orchestra.run.smithi089.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-14T22:47:05.831 INFO:teuthology.orchestra.run.smithi089.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-09-14T22:47:05.856 INFO:teuthology.orchestra.run.smithi089.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-14T22:47:05.881 INFO:teuthology.orchestra.run.smithi089.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-14T22:47:05.886 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:05.886 INFO:teuthology.orchestra.run.smithi089.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-14T22:47:06.038 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:06.081 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:06.081 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:06.394 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:40 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:06.545 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:06.547 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:19.1.1-324.gecab7be5.el9 ceph 6.5 k 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:19.1.1-324.gecab7be5.el9 ceph 2.4 M 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:19.1.1-324.gecab7be5.el9 ceph 1.5 M 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 249 k 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:19.1.1-324.gecab7be5.el9 ceph 4.7 M 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 17 M 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-14T22:47:06.548 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-14T22:47:06.549 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-14T22:47:06.550 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-14T22:47:06.551 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-14T22:47:06.551 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-14T22:47:06.551 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-14T22:47:06.551 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-14T22:47:06.551 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:06.551 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:47:06.551 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:06.551 INFO:teuthology.orchestra.run.smithi052.stdout:Install 37 Packages 2024-09-14T22:47:06.551 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:06.552 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 30 M 2024-09-14T22:47:06.552 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 108 M 2024-09-14T22:47:06.552 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:47:06.568 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:06.568 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:06.973 INFO:teuthology.orchestra.run.smithi052.stdout:(1/37): ceph-19.1.1-324.gecab7be5.el9.x86_64.rp 49 kB/s | 6.5 kB 00:00 2024-09-14T22:47:07.189 INFO:teuthology.orchestra.run.smithi052.stdout:(2/37): ceph-mds-19.1.1-324.gecab7be5.el9.x86_6 6.8 MB/s | 2.4 MB 00:00 2024-09-14T22:47:07.231 INFO:teuthology.orchestra.run.smithi052.stdout:(3/37): ceph-mgr-19.1.1-324.gecab7be5.el9.x86_6 3.9 MB/s | 1.5 MB 00:00 2024-09-14T22:47:07.298 INFO:teuthology.orchestra.run.smithi052.stdout:(4/37): ceph-mgr-modules-core-19.1.1-324.gecab7 3.7 MB/s | 249 kB 00:00 2024-09-14T22:47:07.607 INFO:teuthology.orchestra.run.smithi052.stdout:(5/37): ceph-mon-19.1.1-324.gecab7be5.el9.x86_6 7.4 MB/s | 4.7 MB 00:00 2024-09-14T22:47:07.774 INFO:teuthology.orchestra.run.smithi052.stdout:(6/37): ceph-osd-19.1.1-324.gecab7be5.el9.x86_6 29 MB/s | 17 MB 00:00 2024-09-14T22:47:07.891 INFO:teuthology.orchestra.run.smithi052.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 428 kB/s | 253 kB 00:00 2024-09-14T22:47:07.953 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:08.008 INFO:teuthology.orchestra.run.smithi052.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-14T22:47:08.010 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-14T22:47:08.055 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-14T22:47:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-14T22:47:08.124 INFO:teuthology.orchestra.run.smithi052.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.0 MB/s | 125 kB 00:00 2024-09-14T22:47:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-14T22:47:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-14T22:47:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-14T22:47:08.258 INFO:teuthology.orchestra.run.smithi052.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 220 kB/s | 106 kB 00:00 2024-09-14T22:47:08.283 INFO:teuthology.orchestra.run.smithi052.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-09-14T22:47:08.328 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-14T22:47:08.400 INFO:teuthology.orchestra.run.smithi052.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-09-14T22:47:08.406 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-14T22:47:08.426 INFO:teuthology.orchestra.run.smithi052.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-14T22:47:08.456 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-14T22:47:08.459 INFO:teuthology.orchestra.run.smithi052.stdout:(14/37): python3-cryptography-36.0.1-4.el9.x86_ 1.4 MB/s | 1.2 MB 00:00 2024-09-14T22:47:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-14T22:47:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-14T22:47:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-14T22:47:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-14T22:47:08.601 INFO:teuthology.orchestra.run.smithi052.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 173 kB/s | 35 kB 00:00 2024-09-14T22:47:08.628 INFO:teuthology.orchestra.run.smithi052.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 385 kB/s | 77 kB 00:00 2024-09-14T22:47:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-14T22:47:08.689 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-14T22:47:08.694 INFO:teuthology.orchestra.run.smithi052.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 218 kB/s | 51 kB 00:00 2024-09-14T22:47:08.720 INFO:teuthology.orchestra.run.smithi052.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-09-14T22:47:08.723 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-14T22:47:08.744 INFO:teuthology.orchestra.run.smithi052.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 292 kB/s | 42 kB 00:00 2024-09-14T22:47:08.772 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-14T22:47:08.802 INFO:teuthology.orchestra.run.smithi052.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 521 kB/s | 43 kB 00:00 2024-09-14T22:47:08.812 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-14T22:47:08.828 INFO:teuthology.orchestra.run.smithi052.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-09-14T22:47:08.853 INFO:teuthology.orchestra.run.smithi052.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2024-09-14T22:47:08.878 INFO:teuthology.orchestra.run.smithi052.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 715 kB/s | 18 kB 00:00 2024-09-14T22:47:08.894 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-14T22:47:08.903 INFO:teuthology.orchestra.run.smithi052.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-14T22:47:08.929 INFO:teuthology.orchestra.run.smithi052.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-14T22:47:08.954 INFO:teuthology.orchestra.run.smithi052.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-14T22:47:08.979 INFO:teuthology.orchestra.run.smithi052.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-14T22:47:09.004 INFO:teuthology.orchestra.run.smithi052.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-09-14T22:47:09.030 INFO:teuthology.orchestra.run.smithi052.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-14T22:47:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-14T22:47:09.055 INFO:teuthology.orchestra.run.smithi052.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-14T22:47:09.080 INFO:teuthology.orchestra.run.smithi052.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-14T22:47:09.105 INFO:teuthology.orchestra.run.smithi052.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-14T22:47:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-14T22:47:09.131 INFO:teuthology.orchestra.run.smithi052.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-14T22:47:09.156 INFO:teuthology.orchestra.run.smithi052.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-14T22:47:09.173 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-14T22:47:09.181 INFO:teuthology.orchestra.run.smithi052.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-14T22:47:09.198 INFO:teuthology.orchestra.run.smithi052.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 478 kB/s | 20 kB 00:00 2024-09-14T22:47:09.213 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-14T22:47:09.232 INFO:teuthology.orchestra.run.smithi052.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-14T22:47:09.235 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:09.235 INFO:teuthology.orchestra.run.smithi052.stdout:Total 11 MB/s | 30 MB 00:02 2024-09-14T22:47:09.270 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-14T22:47:09.334 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-14T22:47:09.393 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:47:09.438 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:47:09.439 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:47:09.529 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-14T22:47:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-14T22:47:09.654 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-14T22:47:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-14T22:47:09.809 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-14T22:47:09.926 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-14T22:47:09.940 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:47:09.940 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:47:10.091 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.n 32/37 2024-09-14T22:47:10.121 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 33/37 2024-09-14T22:47:10.146 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 33/37 2024-09-14T22:47:10.146 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:10.146 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-14T22:47:10.146 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T22:47:10.146 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T22:47:10.146 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:11.331 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:47:11.393 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-14T22:47:11.433 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-14T22:47:11.474 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-14T22:47:11.567 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-14T22:47:11.580 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 34/37 2024-09-14T22:47:11.608 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 34/37 2024-09-14T22:47:11.608 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:11.608 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-14T22:47:11.609 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T22:47:11.609 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T22:47:11.609 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:11.753 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-14T22:47:11.793 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-14T22:47:11.887 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-14T22:47:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-14T22:47:12.017 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-14T22:47:12.052 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-14T22:47:12.056 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 35/37 2024-09-14T22:47:12.080 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 35/37 2024-09-14T22:47:12.080 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:12.080 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-14T22:47:12.080 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T22:47:12.080 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T22:47:12.080 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:12.081 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-14T22:47:12.116 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-14T22:47:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-14T22:47:12.191 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-14T22:47:12.234 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-14T22:47:12.268 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-14T22:47:12.316 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 36/37 2024-09-14T22:47:12.317 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-14T22:47:12.339 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 36/37 2024-09-14T22:47:12.340 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:12.340 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-14T22:47:12.340 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T22:47:12.340 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T22:47:12.340 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:12.357 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-14T22:47:12.403 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:19.1.1-324.gecab7be5.el9.x86_64 37/37 2024-09-14T22:47:12.434 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-14T22:47:12.595 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-14T22:47:12.684 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-14T22:47:12.861 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-14T22:47:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-14T22:47:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-14T22:47:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-14T22:47:13.217 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-14T22:47:13.284 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-14T22:47:13.365 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-14T22:47:13.456 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-14T22:47:13.499 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-14T22:47:13.626 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-14T22:47:13.795 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.n 32/37 2024-09-14T22:47:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 33/37 2024-09-14T22:47:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 33/37 2024-09-14T22:47:13.883 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:13.883 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-14T22:47:13.883 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T22:47:13.883 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T22:47:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:14.293 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-2:19.1.1-324.gecab7be5.el9.x86_64 37/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:19.1.1-324.gecab7be5.el9.x86_64 1/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 2/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 3/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 4/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 5/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.n 6/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-14T22:47:14.294 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-14T22:47:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-14T22:47:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-14T22:47:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-14T22:47:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-14T22:47:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-14T22:47:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-14T22:47:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-14T22:47:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-14T22:47:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-14T22:47:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-14T22:47:15.176 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-14T22:47:15.176 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:15.176 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:15.176 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:15.176 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:15.176 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:15.176 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: lua-5.4.4-4.el9.x86_64 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-14T22:47:15.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-14T22:47:15.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-14T22:47:15.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-14T22:47:15.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-14T22:47:15.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-14T22:47:15.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-14T22:47:15.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-14T22:47:15.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-14T22:47:15.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-14T22:47:15.179 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:15.179 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:15.384 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 34/37 2024-09-14T22:47:15.413 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 34/37 2024-09-14T22:47:15.413 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:15.413 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-14T22:47:15.413 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T22:47:15.413 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T22:47:15.413 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:15.548 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-base 2024-09-14T22:47:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 35/37 2024-09-14T22:47:15.902 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 35/37 2024-09-14T22:47:15.902 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:15.902 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-14T22:47:15.902 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T22:47:15.902 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T22:47:15.902 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:16.119 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:52 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 36/37 2024-09-14T22:47:16.177 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 36/37 2024-09-14T22:47:16.177 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:16.177 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-14T22:47:16.178 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T22:47:16.178 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T22:47:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:16.209 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:47:16.244 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:19.1.1-324.gecab7be5.el9.x86_64 37/37 2024-09-14T22:47:16.251 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:16.252 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:47:16.252 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:16.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install cephadm 2024-09-14T22:47:16.894 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:52 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:17.026 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:17.026 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 765 k 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 765 k 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 771 k 2024-09-14T22:47:17.027 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:17.292 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm-19.1.1-324.gecab7be5.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-09-14T22:47:17.293 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:17.293 INFO:teuthology.orchestra.run.smithi089.stdout:Total 2.8 MB/s | 765 kB 00:00 2024-09-14T22:47:17.293 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:17.298 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:17.298 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:17.304 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:17.304 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:17.376 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:17.852 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:17.929 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:19.1.1-324.gecab7be5.el9.x86_64 37/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:19.1.1-324.gecab7be5.el9.x86_64 1/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 2/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 3/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 4/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 5/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.n 6/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-14T22:47:17.995 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-14T22:47:17.996 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-14T22:47:17.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-14T22:47:18.271 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:18.495 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:18.495 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:18.495 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:18.496 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:18.496 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:18.496 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:18.750 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-immutable-object-cache 2024-09-14T22:47:18.931 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-14T22:47:18.931 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: lua-5.4.4-4.el9.x86_64 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-14T22:47:18.932 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-14T22:47:18.933 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:18.934 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:19.324 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2024-09-14T22:47:19.327 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:55 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:19.461 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-324.gecab7be5.el9 ceph 145 k 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:19.462 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 145 k 2024-09-14T22:47:19.463 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 435 k 2024-09-14T22:47:19.463 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:19.676 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-immutable-object-cache-19.1.1-324.gecab7be 682 kB/s | 145 kB 00:00 2024-09-14T22:47:19.676 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:19.677 INFO:teuthology.orchestra.run.smithi089.stdout:Total 676 kB/s | 145 kB 00:00 2024-09-14T22:47:19.677 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:19.684 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:19.685 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:19.741 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:19.741 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:19.887 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:19.922 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:53 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:19.981 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-14T22:47:20.002 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-14T22:47:20.002 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:20.002 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-14T22:47:20.002 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:20.016 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:47:20.060 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:20.061 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:47:20.061 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:20.116 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2024-09-14T22:47:20.705 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:54 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:20.844 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 765 k 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:20.845 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 765 k 2024-09-14T22:47:20.846 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 771 k 2024-09-14T22:47:20.846 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:47:20.921 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-14T22:47:20.921 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:20.921 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:20.921 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:20.921 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:20.922 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:21.084 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-19.1.1-324.gecab7be5.el9.noarch.rpm 3.1 MB/s | 765 kB 00:00 2024-09-14T22:47:21.085 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:21.085 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.1 MB/s | 765 kB 00:00 2024-09-14T22:47:21.085 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:47:21.090 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:47:21.091 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:47:21.096 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:47:21.097 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:47:21.145 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr 2024-09-14T22:47:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:47:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:21.698 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:21.716 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:57 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:21.809 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:47:21.853 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:21.854 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:47:21.854 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:21.943 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-dashboard 2024-09-14T22:47:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:22.446 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:22.446 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:22.446 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:22.447 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:22.502 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:58 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:22.634 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:22.635 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:47:22.635 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 3.6 M 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 32 k 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 16 k 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-14T22:47:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-14T22:47:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-14T22:47:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-14T22:47:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-14T22:47:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:22.637 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:22.637 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:22.637 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2024-09-14T22:47:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:22.638 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 8.7 M 2024-09-14T22:47:22.638 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 104 M 2024-09-14T22:47:22.638 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:22.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2024-09-14T22:47:23.041 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): ceph-grafana-dashboards-19.1.1-324.geca 205 kB/s | 32 kB 00:00 2024-09-14T22:47:23.066 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): ceph-prometheus-alerts-19.1.1-324.gecab 91 kB/s | 16 kB 00:00 2024-09-14T22:47:23.235 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:57 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:23.266 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-14T22:47:23.367 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): ceph-mgr-dashboard-19.1.1-324.gecab7be5 7.4 MB/s | 3.6 MB 00:00 2024-09-14T22:47:23.373 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:23.373 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:23.373 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-14T22:47:23.373 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:23.373 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:47:23.373 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-324.gecab7be5.el9 ceph 145 k 2024-09-14T22:47:23.374 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:23.374 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:47:23.374 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:23.374 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-14T22:47:23.374 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:23.374 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 145 k 2024-09-14T22:47:23.374 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 435 k 2024-09-14T22:47:23.374 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:47:23.417 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-14T22:47:23.442 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 775 kB/s | 19 kB 00:00 2024-09-14T22:47:23.501 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-09-14T22:47:23.526 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.1 MB/s | 1.0 MB 00:00 2024-09-14T22:47:23.552 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-09-14T22:47:23.557 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-19.1.1-324.gecab7be 793 kB/s | 145 kB 00:00 2024-09-14T22:47:23.558 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:23.558 INFO:teuthology.orchestra.run.smithi052.stdout:Total 788 kB/s | 145 kB 00:00 2024-09-14T22:47:23.558 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:47:23.566 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:47:23.566 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:47:23.585 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.0 MB/s | 188 kB 00:00 2024-09-14T22:47:23.610 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 366 kB/s | 31 kB 00:00 2024-09-14T22:47:23.626 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:47:23.627 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:47:23.776 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:47:23.825 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-14T22:47:23.851 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-14T22:47:23.851 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:23.851 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-14T22:47:23.852 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:24.003 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.1 MB/s | 863 kB 00:00 2024-09-14T22:47:24.006 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:24.006 INFO:teuthology.orchestra.run.smithi089.stdout:Total 6.3 MB/s | 8.7 MB 00:01 2024-09-14T22:47:24.055 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:24.071 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:24.071 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:24.287 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:24.287 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:24.845 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:24.920 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.e 1/1 2024-09-14T22:47:24.921 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:24.921 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:24.921 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:24.921 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:24.921 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:24.936 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-14T22:47:24.980 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-14T22:47:25.027 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-14T22:47:25.066 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-14T22:47:25.118 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2024-09-14T22:47:25.166 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-14T22:47:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-14T22:47:25.386 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-14T22:47:25.429 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-14T22:47:25.467 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-14T22:47:25.498 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9. 10/12 2024-09-14T22:47:25.706 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:59 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:25.800 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:47:25.845 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:25.846 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:47:25.846 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:25.907 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2024-09-14T22:47:26.492 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:00 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:26.548 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9 11/12 2024-09-14T22:47:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 12/12 2024-09-14T22:47:26.632 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 3.6 M 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 32 k 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 16 k 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-14T22:47:26.633 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-09-14T22:47:26.634 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:26.635 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 8.7 M 2024-09-14T22:47:26.635 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 104 M 2024-09-14T22:47:26.635 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:47:27.269 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): ceph-grafana-dashboards-19.1.1-324.geca 197 kB/s | 32 kB 00:00 2024-09-14T22:47:27.294 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): ceph-prometheus-alerts-19.1.1-324.gecab 88 kB/s | 16 kB 00:00 2024-09-14T22:47:27.436 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-14T22:47:27.487 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-09-14T22:47:27.537 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-14T22:47:27.562 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-09-14T22:47:27.621 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): ceph-mgr-dashboard-19.1.1-324.gecab7be5 6.9 MB/s | 3.6 MB 00:00 2024-09-14T22:47:27.671 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-09-14T22:47:27.688 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-14T22:47:27.713 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-14T22:47:27.751 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 12/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9 1/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 2/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9. 3/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-14T22:47:27.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-14T22:47:27.755 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.6 MB/s | 863 kB 00:00 2024-09-14T22:47:27.805 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.4 MB/s | 2.0 MB 00:00 2024-09-14T22:47:27.809 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:27.810 INFO:teuthology.orchestra.run.smithi052.stdout:Total 7.4 MB/s | 8.7 MB 00:01 2024-09-14T22:47:27.862 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:47:27.881 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:47:27.881 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:47:28.061 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-14T22:47:28.061 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:28.061 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:28.061 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-14T22:47:28.061 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:28.061 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:28.061 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:28.062 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:28.100 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:47:28.101 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:47:28.324 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-14T22:47:28.655 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:47:28.715 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-14T22:47:28.758 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-14T22:47:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-14T22:47:28.843 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-14T22:47:28.888 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:04 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:28.949 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-14T22:47:29.024 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:29.025 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 7.4 M 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-14T22:47:29.026 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-14T22:47:29.027 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-14T22:47:29.027 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-14T22:47:29.027 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-14T22:47:29.027 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-14T22:47:29.027 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:29.027 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:29.027 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-09-14T22:47:29.027 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2024-09-14T22:47:29.027 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:29.028 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 38 M 2024-09-14T22:47:29.028 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 212 M 2024-09-14T22:47:29.028 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:29.084 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-14T22:47:29.185 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-14T22:47:29.232 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-14T22:47:29.259 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-14T22:47:29.291 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9. 10/12 2024-09-14T22:47:29.363 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-14T22:47:29.396 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 801 kB 00:00 2024-09-14T22:47:29.772 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): ceph-mgr-diskprediction-local-19.1.1-32 13 MB/s | 7.4 MB 00:00 2024-09-14T22:47:29.964 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-09-14T22:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-14T22:47:30.419 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9 11/12 2024-09-14T22:47:30.458 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 12/12 2024-09-14T22:47:30.514 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-09-14T22:47:30.564 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 64 kB/s | 38 kB 00:00 2024-09-14T22:47:31.273 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 326 kB/s | 245 kB 00:00 2024-09-14T22:47:31.449 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.1 MB/s | 4.9 MB 00:01 2024-09-14T22:47:31.811 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 12/12 2024-09-14T22:47:31.812 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9 1/12 2024-09-14T22:47:31.812 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noar 2/12 2024-09-14T22:47:31.812 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9. 3/12 2024-09-14T22:47:31.812 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-14T22:47:31.812 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-14T22:47:31.812 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-14T22:47:31.812 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-14T22:47:31.813 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-14T22:47:31.813 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-14T22:47:31.813 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-14T22:47:31.813 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-14T22:47:31.840 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-14T22:47:32.132 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 292 kB/s | 250 kB 00:00 2024-09-14T22:47:32.136 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-14T22:47:32.136 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:32.136 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:32.136 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-14T22:47:32.136 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:32.136 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:32.136 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:32.137 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:32.342 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-14T22:47:32.900 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-14T22:47:32.901 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:32.901 INFO:teuthology.orchestra.run.smithi089.stdout:Total 9.9 MB/s | 38 MB 00:03 2024-09-14T22:47:32.930 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:06 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:33.070 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:33.071 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================== 2024-09-14T22:47:33.071 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:47:33.071 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================== 2024-09-14T22:47:33.071 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:47:33.071 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 7.4 M 2024-09-14T22:47:33.071 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-14T22:47:33.071 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-14T22:47:33.071 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================== 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-09-14T22:47:33.072 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:33.073 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 38 M 2024-09-14T22:47:33.073 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 212 M 2024-09-14T22:47:33.073 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:47:33.262 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:33.288 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:33.288 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:33.715 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:33.715 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:33.954 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): ceph-mgr-diskprediction-local-19.1.1-32 15 MB/s | 7.4 MB 00:00 2024-09-14T22:47:34.071 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 33 kB 00:00 2024-09-14T22:47:34.229 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 245 kB/s | 191 kB 00:00 2024-09-14T22:47:34.288 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-09-14T22:47:34.355 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 579 kB/s | 38 kB 00:00 2024-09-14T22:47:34.388 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 854 kB/s | 801 kB 00:00 2024-09-14T22:47:34.414 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 99 kB/s | 18 kB 00:00 2024-09-14T22:47:34.483 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:34.545 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 26 MB/s | 4.9 MB 00:00 2024-09-14T22:47:34.566 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-14T22:47:34.615 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.6 MB/s | 250 kB 00:00 2024-09-14T22:47:34.630 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-14T22:47:34.640 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 977 kB/s | 245 kB 00:00 2024-09-14T22:47:34.738 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-14T22:47:34.782 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-14T22:47:34.923 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 53 MB/s | 16 MB 00:00 2024-09-14T22:47:34.971 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-14T22:47:34.990 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.9 MB/s | 5.1 MB 00:00 2024-09-14T22:47:34.991 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:34.991 INFO:teuthology.orchestra.run.smithi052.stdout:Total 20 MB/s | 38 MB 00:01 2024-09-14T22:47:34.998 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-14T22:47:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-14T22:47:35.365 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:47:35.393 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:47:35.393 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:47:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-14T22:47:35.836 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-14T22:47:35.848 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:47:35.849 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:47:36.640 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:47:36.706 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-14T22:47:36.750 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-14T22:47:36.852 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-14T22:47:36.889 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-14T22:47:37.058 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T22:47:37.076 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-14T22:47:37.113 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-14T22:47:37.193 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-14T22:47:37.884 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-14T22:47:38.005 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-14T22:47:38.096 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-14T22:47:38.125 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 12/12 2024-09-14T22:47:39.320 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T22:47:40.403 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-14T22:47:40.428 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 12/12 2024-09-14T22:47:40.801 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 12/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 1/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T22:47:40.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-14T22:47:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-14T22:47:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-14T22:47:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-14T22:47:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-14T22:47:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:41.206 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:41.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-rook 2024-09-14T22:47:42.228 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:18 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:42.362 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:42.363 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:42.363 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:47:42.363 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:42.363 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:42.363 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 49 k 2024-09-14T22:47:42.363 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-14T22:47:42.363 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-14T22:47:42.363 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-14T22:47:42.363 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:42.364 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:42.365 INFO:teuthology.orchestra.run.smithi089.stdout:Install 13 Packages 2024-09-14T22:47:42.365 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:42.365 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.2 M 2024-09-14T22:47:42.365 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 26 M 2024-09-14T22:47:42.365 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:42.546 INFO:teuthology.orchestra.run.smithi089.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 254 kB/s | 19 kB 00:00 2024-09-14T22:47:42.563 INFO:teuthology.orchestra.run.smithi089.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 278 kB/s | 26 kB 00:00 2024-09-14T22:47:42.639 INFO:teuthology.orchestra.run.smithi089.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-09-14T22:47:42.664 INFO:teuthology.orchestra.run.smithi089.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-09-14T22:47:42.698 INFO:teuthology.orchestra.run.smithi089.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.6 MB/s | 279 kB 00:00 2024-09-14T22:47:42.715 INFO:teuthology.orchestra.run.smithi089.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-14T22:47:42.732 INFO:teuthology.orchestra.run.smithi089.stdout:(7/13): ceph-mgr-rook-19.1.1-324.gecab7be5.el9. 189 kB/s | 49 kB 00:00 2024-09-14T22:47:42.782 INFO:teuthology.orchestra.run.smithi089.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 384 kB/s | 32 kB 00:00 2024-09-14T22:47:42.807 INFO:teuthology.orchestra.run.smithi089.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 153 kB/s | 14 kB 00:00 2024-09-14T22:47:42.833 INFO:teuthology.orchestra.run.smithi089.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-14T22:47:42.858 INFO:teuthology.orchestra.run.smithi089.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-14T22:47:42.883 INFO:teuthology.orchestra.run.smithi089.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-14T22:47:42.917 INFO:teuthology.orchestra.run.smithi089.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-14T22:47:42.920 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:42.920 INFO:teuthology.orchestra.run.smithi089.stdout:Total 4.0 MB/s | 2.2 MB 00:00 2024-09-14T22:47:42.985 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:43.000 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:43.000 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:43.075 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 12/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7b 1/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T22:47:43.076 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-14T22:47:43.087 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:43.089 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:43.538 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-14T22:47:43.538 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:43.538 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:43.538 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:43.538 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-14T22:47:43.538 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-14T22:47:43.538 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:43.539 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:43.590 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:43.758 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-14T22:47:43.818 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-14T22:47:43.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2024-09-14T22:47:43.874 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-14T22:47:43.911 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-14T22:47:43.953 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-14T22:47:44.064 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-14T22:47:44.174 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-14T22:47:44.230 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-14T22:47:44.457 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:18 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:44.597 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:44.598 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:44.598 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:47:44.598 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:44.598 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:47:44.598 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 49 k 2024-09-14T22:47:44.598 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-14T22:47:44.598 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-14T22:47:44.598 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:44.599 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2024-09-14T22:47:44.600 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:44.600 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.2 M 2024-09-14T22:47:44.600 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2024-09-14T22:47:44.600 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:47:44.954 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 174 kB/s | 26 kB 00:00 2024-09-14T22:47:44.956 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-14T22:47:44.979 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 19 kB 00:00 2024-09-14T22:47:45.004 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): ceph-mgr-rook-19.1.1-324.gecab7be5.el9. 250 kB/s | 49 kB 00:00 2024-09-14T22:47:45.020 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-14T22:47:45.053 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-14T22:47:45.101 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-14T22:47:45.123 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 13/13 2024-09-14T22:47:45.213 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 680 kB/s | 159 kB 00:00 2024-09-14T22:47:45.238 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 859 kB/s | 222 kB 00:00 2024-09-14T22:47:45.271 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-09-14T22:47:45.305 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-14T22:47:45.330 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 855 kB/s | 279 kB 00:00 2024-09-14T22:47:45.355 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 172 kB/s | 14 kB 00:00 2024-09-14T22:47:45.389 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-14T22:47:45.414 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-14T22:47:45.440 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-14T22:47:45.473 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-14T22:47:45.475 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:45.476 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-14T22:47:45.541 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:47:45.557 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:47:45.557 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:47:45.653 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:47:45.655 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:47:45.904 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 13/13 2024-09-14T22:47:45.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 1/13 2024-09-14T22:47:45.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-14T22:47:45.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-14T22:47:45.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-14T22:47:45.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-14T22:47:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-14T22:47:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-14T22:47:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-14T22:47:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-14T22:47:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-14T22:47:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-14T22:47:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-14T22:47:46.175 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:47:46.242 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-14T22:47:46.242 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-14T22:47:46.243 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-14T22:47:46.244 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-14T22:47:46.244 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:46.244 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:46.353 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-14T22:47:46.476 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-14T22:47:46.500 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-cephadm 2024-09-14T22:47:46.533 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-14T22:47:46.567 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-14T22:47:46.609 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-14T22:47:46.740 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-14T22:47:46.852 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-14T22:47:46.903 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-14T22:47:47.069 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:23 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:47.204 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:47.204 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 146 k 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:47.205 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:47.206 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:47.206 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-09-14T22:47:47.206 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:47.206 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 7.0 M 2024-09-14T22:47:47.206 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 33 M 2024-09-14T22:47:47.207 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:47.497 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-14T22:47:47.530 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): ceph-mgr-cephadm-19.1.1-324.gecab7be5.el 685 kB/s | 146 kB 00:00 2024-09-14T22:47:47.589 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-14T22:47:47.614 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 689 kB/s | 58 kB 00:00 2024-09-14T22:47:47.640 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-14T22:47:47.667 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-14T22:47:47.716 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-14T22:47:47.750 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-14T22:47:47.798 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-14T22:47:47.827 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 13/13 2024-09-14T22:47:48.598 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 13/13 2024-09-14T22:47:48.598 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 1/13 2024-09-14T22:47:48.598 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-14T22:47:48.599 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-14T22:47:48.756 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.1 MB/s | 6.0 MB 00:01 2024-09-14T22:47:48.759 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:48.759 INFO:teuthology.orchestra.run.smithi089.stdout:Total 4.5 MB/s | 7.0 MB 00:01 2024-09-14T22:47:48.830 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:48.841 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:48.841 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:48.926 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-14T22:47:48.926 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:48.927 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:48.927 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:48.927 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-14T22:47:48.927 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-14T22:47:48.927 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-14T22:47:48.927 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-14T22:47:48.928 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-14T22:47:48.928 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-14T22:47:48.928 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-14T22:47:48.928 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-14T22:47:48.928 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-14T22:47:48.928 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-14T22:47:48.929 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-14T22:47:48.929 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-14T22:47:48.929 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:48.929 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:48.941 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:48.943 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:49.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2024-09-14T22:47:49.253 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-14T22:47:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-14T22:47:49.784 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:23 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:49.924 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:49.925 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:49.925 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:47:49.925 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:49.925 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:47:49.925 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 146 k 2024-09-14T22:47:49.925 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-14T22:47:49.925 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-14T22:47:49.925 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-14T22:47:49.925 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-14T22:47:49.926 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-14T22:47:49.926 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-14T22:47:49.926 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:49.926 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:47:49.926 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:49.926 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-09-14T22:47:49.926 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:49.927 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 7.0 M 2024-09-14T22:47:49.927 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 33 M 2024-09-14T22:47:49.927 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:47:50.003 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-14T22:47:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-14T22:47:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-14T22:47:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 6/6 2024-09-14T22:47:50.449 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-14T22:47:50.475 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): ceph-mgr-cephadm-19.1.1-324.gecab7be5.el 611 kB/s | 146 kB 00:00 2024-09-14T22:47:50.525 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-14T22:47:50.550 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.3 MB/s | 548 kB 00:00 2024-09-14T22:47:50.575 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-14T22:47:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 6/6 2024-09-14T22:47:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/6 2024-09-14T22:47:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-14T22:47:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-14T22:47:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-14T22:47:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-14T22:47:51.017 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.6 MB/s | 6.0 MB 00:00 2024-09-14T22:47:51.020 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:51.020 INFO:teuthology.orchestra.run.smithi052.stdout:Total 6.4 MB/s | 7.0 MB 00:01 2024-09-14T22:47:51.094 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:47:51.106 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:47:51.106 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:47:51.212 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:47:51.213 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:47:51.251 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-14T22:47:51.251 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:51.251 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:51.251 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:51.251 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-14T22:47:51.251 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-14T22:47:51.251 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-14T22:47:51.251 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-14T22:47:51.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-14T22:47:51.252 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:51.252 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:51.515 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-fuse 2024-09-14T22:47:51.549 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:47:51.726 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-14T22:47:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-14T22:47:52.076 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:28 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:52.212 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 896 k 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:52.213 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 976 k 2024-09-14T22:47:52.214 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 2.9 M 2024-09-14T22:47:52.214 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-14T22:47:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-14T22:47:52.502 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 512 kB/s | 80 kB 00:00 2024-09-14T22:47:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-14T22:47:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 6/6 2024-09-14T22:47:52.852 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): ceph-fuse-19.1.1-324.gecab7be5.el9.x86_6 1.7 MB/s | 896 kB 00:00 2024-09-14T22:47:52.853 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:52.853 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.5 MB/s | 976 kB 00:00 2024-09-14T22:47:52.865 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:52.888 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:52.888 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:52.972 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:52.972 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:53.168 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:53.312 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-14T22:47:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 6/6 2024-09-14T22:47:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 1/6 2024-09-14T22:47:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-14T22:47:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-14T22:47:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-14T22:47:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-14T22:47:53.393 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2/2 2024-09-14T22:47:53.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:53.610 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:53.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2024-09-14T22:47:54.081 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2/2 2024-09-14T22:47:54.081 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/2 2024-09-14T22:47:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-14T22:47:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:54.342 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-14T22:47:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:54.342 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:54.453 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:28 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:54.594 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:54.594 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:54.594 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:47:54.594 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:54.594 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:47:54.594 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 896 k 2024-09-14T22:47:54.595 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-14T22:47:54.595 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-14T22:47:54.595 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:54.595 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:47:54.595 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:54.595 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-09-14T22:47:54.595 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:54.596 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 976 k 2024-09-14T22:47:54.596 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.9 M 2024-09-14T22:47:54.596 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:47:54.657 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-volume 2024-09-14T22:47:54.835 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 566 kB/s | 80 kB 00:00 2024-09-14T22:47:54.969 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-fuse-19.1.1-324.gecab7be5.el9.x86_6 3.2 MB/s | 896 kB 00:00 2024-09-14T22:47:54.969 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:54.970 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.5 MB/s | 976 kB 00:00 2024-09-14T22:47:54.982 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:47:55.004 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:47:55.004 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:47:55.090 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:47:55.090 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:47:55.240 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:31 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:55.282 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:47:55.376 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:55.377 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:55.377 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-14T22:47:55.377 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:55.377 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:55.377 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 285 k 2024-09-14T22:47:55.377 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:55.377 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:55.378 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:55.378 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-14T22:47:55.378 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:55.378 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 285 k 2024-09-14T22:47:55.378 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 1.3 M 2024-09-14T22:47:55.378 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:55.417 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-14T22:47:55.488 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2/2 2024-09-14T22:47:55.618 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-volume-19.1.1-324.gecab7be5.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-14T22:47:55.619 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:55.619 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-14T22:47:55.619 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:55.628 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:55.628 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:55.660 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:55.661 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:55.889 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:56.005 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:56.021 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:56.021 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-14T22:47:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:56.177 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2/2 2024-09-14T22:47:56.177 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/2 2024-09-14T22:47:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-14T22:47:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:56.476 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-14T22:47:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:56.477 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:56.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-volume 2024-09-14T22:47:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:56.998 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:56.998 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:57.333 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados-devel 2024-09-14T22:47:57.399 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:31 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:47:57.542 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:47:57.543 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:57.543 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-14T22:47:57.543 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:57.543 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:47:57.543 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume noarch 2:19.1.1-324.gecab7be5.el9 ceph-noarch 285 k 2024-09-14T22:47:57.543 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:57.543 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:47:57.543 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:47:57.543 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-14T22:47:57.544 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:57.544 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 285 k 2024-09-14T22:47:57.544 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.3 M 2024-09-14T22:47:57.544 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:47:57.780 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-volume-19.1.1-324.gecab7be5.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-14T22:47:57.781 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:57.781 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-14T22:47:57.781 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:47:57.791 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:47:57.791 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:47:57.821 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:47:57.821 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:47:57.911 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:33 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:47:58.047 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:47:58.048 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:58.049 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-09-14T22:47:58.049 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:58.049 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:47:58.049 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:19.1.1-324.gecab7be5.el9 ceph 127 k 2024-09-14T22:47:58.049 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:58.049 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:47:58.049 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:47:58.049 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-14T22:47:58.050 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:58.050 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 127 k 2024-09-14T22:47:58.050 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 456 k 2024-09-14T22:47:58.050 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:47:58.057 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:47:58.133 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:58.150 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:58.150 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:47:58.150 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-14T22:47:58.150 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:58.300 INFO:teuthology.orchestra.run.smithi089.stdout:librados-devel-19.1.1-324.gecab7be5.el9.x86_64. 505 kB/s | 127 kB 00:00 2024-09-14T22:47:58.301 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:47:58.301 INFO:teuthology.orchestra.run.smithi089.stdout:Total 502 kB/s | 127 kB 00:00 2024-09-14T22:47:58.301 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:47:58.307 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:47:58.307 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:47:58.370 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:47:58.371 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:47:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:47:58.632 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:47:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:47:59.209 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 1/1 2024-09-14T22:47:59.209 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:59.210 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:47:59.210 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume-2:19.1.1-324.gecab7be5.el9.noarch 2024-09-14T22:47:59.210 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:47:59.210 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:47:59.313 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:47:59.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:59.313 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:47:59.313 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:47:59.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:47:59.313 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:47:59.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2024-09-14T22:47:59.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs2 2024-09-14T22:48:00.094 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:34 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:00.183 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:36 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:00.234 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:19.1.1-324.gecab7be5.el9 ceph 127 k 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-14T22:48:00.235 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:00.236 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 127 k 2024-09-14T22:48:00.236 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 456 k 2024-09-14T22:48:00.236 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:48:00.276 INFO:teuthology.orchestra.run.smithi089.stdout:Package libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:00.320 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:00.321 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:48:00.321 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:00.404 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs-devel 2024-09-14T22:48:00.443 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-19.1.1-324.gecab7be5.el9.x86_64. 614 kB/s | 127 kB 00:00 2024-09-14T22:48:00.443 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:48:00.444 INFO:teuthology.orchestra.run.smithi052.stdout:Total 610 kB/s | 127 kB 00:00 2024-09-14T22:48:00.444 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:48:00.449 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:48:00.450 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:48:00.518 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:48:00.519 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:48:00.672 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:48:00.786 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:00.973 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:36 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:01.106 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:19.1.1-324.gecab7be5.el9 ceph 32 k 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-14T22:48:01.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:01.108 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 32 k 2024-09-14T22:48:01.108 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 142 k 2024-09-14T22:48:01.108 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:48:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:01.257 INFO:teuthology.orchestra.run.smithi089.stdout:libcephfs-devel-19.1.1-324.gecab7be5.el9.x86_64 213 kB/s | 32 kB 00:00 2024-09-14T22:48:01.257 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:48:01.257 INFO:teuthology.orchestra.run.smithi089.stdout:Total 211 kB/s | 32 kB 00:00 2024-09-14T22:48:01.257 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:48:01.261 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:48:01.261 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:48:01.279 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:48:01.279 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:48:01.365 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:48:01.462 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:01.474 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:01.474 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:01.474 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:48:01.474 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:48:01.475 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:01.475 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:01.687 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2024-09-14T22:48:01.814 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:02.073 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:48:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:48:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:02.073 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:02.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados2 2024-09-14T22:48:02.282 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:36 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:02.377 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:02.423 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:02.424 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:48:02.424 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:02.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2024-09-14T22:48:02.840 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:38 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:02.932 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:02.976 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:02.977 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:48:02.977 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:03.060 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librbd1 2024-09-14T22:48:03.077 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:37 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:03.215 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:03.216 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-14T22:48:03.216 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:03.216 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:48:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:19.1.1-324.gecab7be5.el9 ceph 32 k 2024-09-14T22:48:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:03.216 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:48:03.217 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:03.217 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-14T22:48:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:03.217 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 32 k 2024-09-14T22:48:03.217 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 142 k 2024-09-14T22:48:03.217 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:48:03.372 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-19.1.1-324.gecab7be5.el9.x86_64 206 kB/s | 32 kB 00:00 2024-09-14T22:48:03.372 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:48:03.372 INFO:teuthology.orchestra.run.smithi052.stdout:Total 204 kB/s | 32 kB 00:00 2024-09-14T22:48:03.372 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:48:03.375 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:48:03.375 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:48:03.395 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:48:03.395 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:48:03.491 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:48:03.596 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:03.623 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:39 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:03.712 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:03.756 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:03.757 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:48:03.757 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:03.855 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rados 2024-09-14T22:48:03.944 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:04.207 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:04.207 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:04.207 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:48:04.207 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:48:04.207 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:04.208 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:04.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2024-09-14T22:48:04.422 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:40 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:04.513 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:04.556 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:04.556 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:48:04.557 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:04.645 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rgw 2024-09-14T22:48:04.974 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:38 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:05.069 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:05.115 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:05.116 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:48:05.117 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:05.180 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2024-09-14T22:48:05.218 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:41 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:05.308 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:05.352 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:05.353 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:48:05.353 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:05.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-cephfs 2024-09-14T22:48:05.772 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:39 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:05.865 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:05.911 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:05.912 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:48:05.912 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:05.966 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2024-09-14T22:48:06.006 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:06.096 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:06.139 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:06.140 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:48:06.140 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:06.230 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rbd 2024-09-14T22:48:06.561 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:40 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:06.654 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:06.701 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:06.702 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:48:06.702 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:06.760 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2024-09-14T22:48:06.793 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:06.883 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:06.926 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:06.927 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:48:06.927 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:07.011 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-fuse 2024-09-14T22:48:07.355 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:41 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:07.451 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:07.497 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:07.498 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:48:07.498 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:07.563 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2024-09-14T22:48:07.577 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:43 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:07.712 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:07.713 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-09-14T22:48:07.713 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:07.713 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:48:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 85 k 2024-09-14T22:48:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:07.714 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:48:07.714 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:07.714 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-14T22:48:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:07.714 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 85 k 2024-09-14T22:48:07.714 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 231 k 2024-09-14T22:48:07.714 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:48:07.919 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-fuse-19.1.1-324.gecab7be5.el9.x86_64.rpm 415 kB/s | 85 kB 00:00 2024-09-14T22:48:07.919 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:48:07.920 INFO:teuthology.orchestra.run.smithi089.stdout:Total 413 kB/s | 85 kB 00:00 2024-09-14T22:48:07.920 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:48:07.927 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:48:07.927 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:48:07.990 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:48:07.990 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:48:08.155 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:48:08.249 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:08.295 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:08.296 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:48:08.296 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:08.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2024-09-14T22:48:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:08.883 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:08.883 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:08.883 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:48:08.883 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:48:08.884 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:08.884 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:08.940 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:09.035 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:09.081 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:09.082 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:48:09.082 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:09.147 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2024-09-14T22:48:09.220 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-mirror 2024-09-14T22:48:09.736 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:43 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:09.817 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:45 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:09.877 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:09.878 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:09.878 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-14T22:48:09.878 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:09.878 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:48:09.878 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:19.1.1-324.gecab7be5.el9 ceph 85 k 2024-09-14T22:48:09.878 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:09.878 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:48:09.878 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:09.878 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-14T22:48:09.879 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:09.879 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 85 k 2024-09-14T22:48:09.879 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 231 k 2024-09-14T22:48:09.879 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:48:09.961 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:09.961 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:09.961 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-09-14T22:48:09.961 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.1 M 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.1 M 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 13 M 2024-09-14T22:48:09.962 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:48:10.074 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-19.1.1-324.gecab7be5.el9.x86_64.rpm 437 kB/s | 85 kB 00:00 2024-09-14T22:48:10.075 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:48:10.075 INFO:teuthology.orchestra.run.smithi052.stdout:Total 434 kB/s | 85 kB 00:00 2024-09-14T22:48:10.075 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:48:10.083 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:48:10.083 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:48:10.155 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:48:10.155 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:48:10.313 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:48:10.316 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-mirror-19.1.1-324.gecab7be5.el9.x86_64.rpm 8.8 MB/s | 3.1 MB 00:00 2024-09-14T22:48:10.317 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:48:10.318 INFO:teuthology.orchestra.run.smithi089.stdout:Total 8.7 MB/s | 3.1 MB 00:00 2024-09-14T22:48:10.318 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:48:10.327 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:48:10.327 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:48:10.407 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:48:10.408 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:48:10.415 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:10.791 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:10.835 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:48:10.927 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:10.953 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:10.953 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:48:10.953 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-14T22:48:10.953 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T22:48:10.953 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T22:48:10.953 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:11.074 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:11.074 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:11.074 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:48:11.074 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:48:11.074 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:11.326 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2024-09-14T22:48:11.941 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:45 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:12.072 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:12.072 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:12.072 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:48:12.072 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:48:12.072 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:12.072 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:12.085 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:12.085 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-14T22:48:12.085 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:12.086 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:48:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:19.1.1-324.gecab7be5.el9 ceph 3.1 M 2024-09-14T22:48:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:12.086 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:48:12.086 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:12.086 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-14T22:48:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:12.086 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.1 M 2024-09-14T22:48:12.087 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 13 M 2024-09-14T22:48:12.087 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:48:12.336 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-nbd 2024-09-14T22:48:12.385 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-19.1.1-324.gecab7be5.el9.x86_64.rpm 10 MB/s | 3.1 MB 00:00 2024-09-14T22:48:12.385 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:48:12.386 INFO:teuthology.orchestra.run.smithi052.stdout:Total 10 MB/s | 3.1 MB 00:00 2024-09-14T22:48:12.386 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:48:12.401 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:48:12.401 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:48:12.481 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:48:12.482 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:48:12.907 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:48 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:48:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:13.045 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:13.045 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-09-14T22:48:13.046 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:13.046 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-14T22:48:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 170 k 2024-09-14T22:48:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:13.046 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-14T22:48:13.046 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-14T22:48:13.046 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-14T22:48:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:13.047 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 170 k 2024-09-14T22:48:13.047 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 494 k 2024-09-14T22:48:13.047 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-14T22:48:13.047 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:13.047 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T22:48:13.047 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-14T22:48:13.047 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T22:48:13.047 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T22:48:13.048 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:13.264 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-nbd-19.1.1-324.gecab7be5.el9.x86_64.rpm 786 kB/s | 170 kB 00:00 2024-09-14T22:48:13.264 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:48:13.265 INFO:teuthology.orchestra.run.smithi089.stdout:Total 780 kB/s | 170 kB 00:00 2024-09-14T22:48:13.269 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-14T22:48:13.273 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-14T22:48:13.273 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-14T22:48:13.336 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-14T22:48:13.337 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-14T22:48:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-14T22:48:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:14.198 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:48:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:48:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:14.198 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:14.330 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:14.330 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:14.330 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-14T22:48:14.330 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:48:14.330 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:14.330 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:14.406 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2024-09-14T22:48:14.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-mirror 2024-09-14T22:48:14.999 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:48 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:15.140 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:15.140 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:15.140 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-14T22:48:15.140 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:15.140 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-14T22:48:15.140 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:19.1.1-324.gecab7be5.el9 ceph 170 k 2024-09-14T22:48:15.141 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:15.141 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-14T22:48:15.141 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-14T22:48:15.141 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-14T22:48:15.141 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:15.141 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 170 k 2024-09-14T22:48:15.141 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 494 k 2024-09-14T22:48:15.142 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-14T22:48:15.193 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:51 ago on Sat 14 Sep 2024 10:46:24 PM UTC. 2024-09-14T22:48:15.288 INFO:teuthology.orchestra.run.smithi089.stdout:Package rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:15.330 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-14T22:48:15.331 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-14T22:48:15.332 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-14T22:48:15.372 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-19.1.1-324.gecab7be5.el9.x86_64.rpm 741 kB/s | 170 kB 00:00 2024-09-14T22:48:15.372 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-14T22:48:15.372 INFO:teuthology.orchestra.run.smithi052.stdout:Total 737 kB/s | 170 kB 00:00 2024-09-14T22:48:15.372 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-14T22:48:15.381 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-14T22:48:15.381 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-14T22:48:15.424 DEBUG:teuthology.parallel:result is None 2024-09-14T22:48:15.448 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-14T22:48:15.449 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-14T22:48:15.608 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-14T22:48:15.706 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:16.407 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 1/1 2024-09-14T22:48:16.407 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:16.407 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-14T22:48:16.407 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:19.1.1-324.gecab7be5.el9.x86_64 2024-09-14T22:48:16.407 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:16.407 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:16.655 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2024-09-14T22:48:17.254 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:51 ago on Sat 14 Sep 2024 10:46:26 PM UTC. 2024-09-14T22:48:17.350 INFO:teuthology.orchestra.run.smithi052.stdout:Package rbd-mirror-2:19.1.1-324.gecab7be5.el9.x86_64 is already installed. 2024-09-14T22:48:17.396 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-14T22:48:17.397 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-14T22:48:17.398 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-14T22:48:17.466 DEBUG:teuthology.parallel:result is None 2024-09-14T22:48:17.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-14T22:48:17.609 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-14T22:48:17.631 INFO:teuthology.orchestra.run.smithi052.stdout:19.1.1-324.gecab7be5.el9 2024-09-14T22:48:17.632 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-324.gecab7be5.el9 2024-09-14T22:48:17.632 INFO:teuthology.task.install:The correct ceph version 19.1.1-324.gecab7be5 is installed. 2024-09-14T22:48:17.634 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-14T22:48:17.775 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-14T22:48:17.801 INFO:teuthology.orchestra.run.smithi089.stdout:19.1.1-324.gecab7be5.el9 2024-09-14T22:48:17.801 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-324.gecab7be5.el9 2024-09-14T22:48:17.801 INFO:teuthology.task.install:The correct ceph version 19.1.1-324.gecab7be5 is installed. 2024-09-14T22:48:17.803 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-14T22:48:17.803 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:17.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T22:48:17.829 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:17.829 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T22:48:17.870 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-14T22:48:17.870 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:17.871 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T22:48:17.896 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T22:48:17.963 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:17.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T22:48:17.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T22:48:18.060 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-14T22:48:18.060 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:18.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T22:48:18.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T22:48:18.151 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:18.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T22:48:18.178 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T22:48:18.246 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-14T22:48:18.246 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:18.246 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T22:48:18.272 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T22:48:18.338 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:18.338 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T22:48:18.365 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T22:48:18.432 INFO:teuthology.run_tasks:Running task ceph... 2024-09-14T22:48:18.517 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-14T22:48:18.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2024-09-14T22:48:18.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /var/log/ceph 2024-09-14T22:48:18.545 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-14T22:48:18.546 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-14T22:48:18.589 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-14T22:48:18.616 INFO:tasks.ceph:Creating extra log directories... 2024-09-14T22:48:18.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-14T22:48:18.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-14T22:48:18.695 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-14T22:48:18.695 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-14T22:48:18.696 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7904733', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 4271, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'sleep_before_teardown': 0, 'subset': '97/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'targets': {'smithi052.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPg450sCgqSZFEMi05UZtTnatCArbuCBeU52Co/YZUkC5+nBt0mPDFFs+mOtIM2CF14NE+PwBDTR8HrPFF6sMvg=', 'smithi089.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOphTnuc232wFEHlDEc7DtT8nVVpon/sJ/e1bz9jx+3kNyW0F3b1ZugF+/ty2d3A/yCN1Y97GFGyi2SJGs5K0AM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-14_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-14T22:48:18.696 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-14T22:48:18.721 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-14T22:48:18.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T22:48:18.777 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T22:48:18.803 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:18.803 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T22:48:18.857 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T22:48:18.857 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-09-14T22:48:18.912 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T22:48:18.912 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T22:48:18.912 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-14T22:48:18.912 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T22:48:18.912 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T22:48:18.912 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-14 22:48:13.298335214 +0000 2024-09-14T22:48:18.912 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-14 22:45:45.896026632 +0000 2024-09-14T22:48:18.912 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-14 22:45:45.896026632 +0000 2024-09-14T22:48:18.913 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-14 22:45:45.896026632 +0000 2024-09-14T22:48:18.913 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T22:48:18.976 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-14T22:48:18.976 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-14T22:48:18.976 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.8345e-05 s, 5.2 MB/s 2024-09-14T22:48:18.977 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T22:48:19.034 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-09-14T22:48:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T22:48:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T22:48:19.090 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-14T22:48:19.090 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T22:48:19.090 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T22:48:19.091 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-14 22:48:13.298335214 +0000 2024-09-14T22:48:19.091 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-14 22:45:46.214022681 +0000 2024-09-14T22:48:19.091 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-14 22:45:46.214022681 +0000 2024-09-14T22:48:19.091 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-14 22:45:46.214022681 +0000 2024-09-14T22:48:19.091 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T22:48:19.154 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-14T22:48:19.154 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-14T22:48:19.155 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 8.2988e-05 s, 6.2 MB/s 2024-09-14T22:48:19.156 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T22:48:19.213 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-09-14T22:48:19.269 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T22:48:19.269 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T22:48:19.270 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-14T22:48:19.270 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T22:48:19.270 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T22:48:19.270 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-14 22:48:13.299335201 +0000 2024-09-14T22:48:19.270 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-14 22:45:46.549018520 +0000 2024-09-14T22:48:19.270 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-14 22:45:46.549018520 +0000 2024-09-14T22:48:19.270 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-14 22:45:46.549018520 +0000 2024-09-14T22:48:19.270 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T22:48:19.333 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-14T22:48:19.334 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-14T22:48:19.334 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000103892 s, 4.9 MB/s 2024-09-14T22:48:19.335 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T22:48:19.392 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-09-14T22:48:19.448 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T22:48:19.448 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T22:48:19.448 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-14T22:48:19.448 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T22:48:19.448 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T22:48:19.448 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-14 22:48:13.299335201 +0000 2024-09-14T22:48:19.448 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-14 22:45:46.893014246 +0000 2024-09-14T22:48:19.448 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-14 22:45:46.893014246 +0000 2024-09-14T22:48:19.448 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-14 22:45:46.893014246 +0000 2024-09-14T22:48:19.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T22:48:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-14T22:48:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-14T22:48:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000104523 s, 4.9 MB/s 2024-09-14T22:48:19.514 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T22:48:19.571 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T22:48:19.571 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T22:48:19.571 INFO:tasks.ceph:Generating config... 2024-09-14T22:48:19.572 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-14T22:48:19.572 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-14T22:48:19.572 INFO:tasks.ceph:[client] log to stderr = False 2024-09-14T22:48:19.572 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-14T22:48:19.573 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-14T22:48:19.573 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:19.573 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:19.573 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:19.573 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:19.573 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:19.573 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:19.573 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:19.573 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:19.573 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-14T22:48:19.573 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-14T22:48:19.573 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-14T22:48:19.574 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-14T22:48:19.574 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-14T22:48:19.574 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-14T22:48:19.574 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-14T22:48:19.574 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-14T22:48:19.574 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-14T22:48:19.575 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-14T22:48:19.575 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-14T22:48:19.575 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-14T22:48:19.575 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-14T22:48:19.575 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-14T22:48:19.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-14T22:48:19.654 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/cluster1.keyring 2024-09-14T22:48:19.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-14T22:48:19.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-14T22:48:19.807 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.52')] 2024-09-14T22:48:19.807 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.52', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-14T22:48:19.808 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:19.808 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T22:48:19.863 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.52 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T22:48:19.944 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T22:48:19.945 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: generated fsid 51bdee54-6793-4abd-9100-af548f45d456 2024-09-14T22:48:19.945 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = quincy 2024-09-14T22:48:19.945 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 0 2024-09-14T22:48:19.945 INFO:teuthology.orchestra.run.smithi052.stdout:fsid 51bdee54-6793-4abd-9100-af548f45d456 2024-09-14T22:48:19.946 INFO:teuthology.orchestra.run.smithi052.stdout:last_changed 2024-09-14T22:48:19.943836+0000 2024-09-14T22:48:19.946 INFO:teuthology.orchestra.run.smithi052.stdout:created 2024-09-14T22:48:19.943836+0000 2024-09-14T22:48:19.946 INFO:teuthology.orchestra.run.smithi052.stdout:min_mon_release 17 (quincy) 2024-09-14T22:48:19.946 INFO:teuthology.orchestra.run.smithi052.stdout:election_strategy: 1 2024-09-14T22:48:19.946 INFO:teuthology.orchestra.run.smithi052.stdout:0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-09-14T22:48:19.946 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-14T22:48:19.948 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T22:48:20.005 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 51bdee54-6793-4abd-9100-af548f45d456... 2024-09-14T22:48:20.007 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-14T22:48:20.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-14T22:48:20.090 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2024-09-14T22:48:20.091 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2024-09-14T22:48:20.092 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2024-09-14T22:48:20.093 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = 172.21.15.52 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: mon client directed command retry = 5 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on pool no app = False 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: ms inject socket failures = 5000 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 51bdee54-6793-4abd-9100-af548f45d456 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock profile = high_recovery_ops 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2024-09-14T22:48:20.094 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug trim objects = true 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2024-09-14T22:48:20.095 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore block size = 96636764160 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore compression algorithm = zstd 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore compression mode = aggressive 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore fsck on mount = True 2024-09-14T22:48:20.096 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluefs = 1/20 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluestore = 1/20 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 20 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: debug rocksdb = 4/10 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd full ratio = 0.9 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd nearfull ratio = 0.8 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: osd failsafe full ratio = 0.95 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: osd objectstore = bluestore 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2024-09-14T22:48:20.097 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-09-14T22:48:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: # 1m isn't quite enough 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: mon_down_mkfs_grace = 2m 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_filestore_osds = false 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: debug rbd = 20 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: debug rbd_mirror = 15 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: log to stderr = False 2024-09-14T22:48:20.100 INFO:teuthology.orchestra.run.smithi052.stdout: rbd default features = 125 2024-09-14T22:48:20.101 INFO:teuthology.orchestra.run.smithi052.stdout: rbd mirror image policy type = simple 2024-09-14T22:48:20.101 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2024-09-14T22:48:20.101 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.0] 2024-09-14T22:48:20.101 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:20.101 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:20.101 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.1] 2024-09-14T22:48:20.101 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:20.101 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:20.101 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.2] 2024-09-14T22:48:20.102 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:20.102 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:20.102 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.3] 2024-09-14T22:48:20.102 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:20.102 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-09-14T22:48:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = 172.21.15.52 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: mon client directed command retry = 5 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on pool no app = False 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject socket failures = 5000 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 51bdee54-6793-4abd-9100-af548f45d456 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-09-14T22:48:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug trim objects = true 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-09-14T22:48:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-09-14T22:48:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore compression algorithm = zstd 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore compression mode = aggressive 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-09-14T22:48:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-09-14T22:48:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T22:48:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: debug rbd = 20 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: debug rbd_mirror = 15 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: log to stderr = False 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: rbd default features = 125 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: rbd mirror image policy type = simple 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.0] 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.1] 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.2] 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:20.113 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.3] 2024-09-14T22:48:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:20.115 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-14T22:48:20.116 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-14T22:48:20.210 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-14T22:48:20.210 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:20.210 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-14T22:48:20.227 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:20.227 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-14T22:48:20.283 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T22:48:20.283 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:20.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-14T22:48:20.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-14T22:48:20.359 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:20.359 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-14T22:48:20.415 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T22:48:20.415 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:20.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-14T22:48:20.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-14T22:48:20.455 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:20.455 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-14T22:48:20.512 INFO:tasks.ceph:Setting up mon nodes... 2024-09-14T22:48:20.512 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-14T22:48:20.512 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-14T22:48:20.564 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-14T22:48:20.567 INFO:tasks.ceph:Setting up mds nodes... 2024-09-14T22:48:20.567 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-14T22:48:20.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-14T22:48:20.609 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-14T22:48:20.623 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-14T22:48:20.677 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-14T22:48:20.690 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-14T22:48:20.731 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-14T22:48:20.743 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-14T22:48:20.824 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-14T22:48:20.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-14T22:48:20.919 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-14T22:48:20.931 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-14T22:48:21.012 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-14T22:48:21.024 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-14T22:48:21.105 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-14T22:48:21.117 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-14T22:48:21.199 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-14T22:48:21.212 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-14T22:48:21.293 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-14T22:48:21.306 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-14T22:48:21.306 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T22:48:21.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-14T22:48:21.334 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T22:48:21.334 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-14T22:48:21.334 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T22:48:21.335 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-14T22:48:21.402 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T22:48:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T22:48:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T22:48:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T22:48:21.402 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T22:48:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-09-14T22:48:21.403 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T22:48:21.403 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T22:48:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T22:48:21.403 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T22:48:21.435 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-09-14T22:48:21.437 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-09-14T22:48:21.437 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-14T22:48:21.509 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-14T22:48:21.580 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-14T22:48:21.647 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T22:48:21.647 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-14T22:48:21.647 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T22:48:21.648 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-14T22:48:21.712 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T22:48:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T22:48:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T22:48:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T22:48:21.712 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T22:48:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-09-14T22:48:21.713 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T22:48:21.713 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T22:48:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T22:48:21.713 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T22:48:21.744 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-09-14T22:48:21.746 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-09-14T22:48:21.746 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-14T22:48:21.816 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-14T22:48:21.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-14T22:48:21.953 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T22:48:21.954 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-14T22:48:21.954 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T22:48:21.954 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-14T22:48:22.019 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T22:48:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T22:48:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T22:48:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T22:48:22.019 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T22:48:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-09-14T22:48:22.019 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T22:48:22.019 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T22:48:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T22:48:22.020 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T22:48:22.051 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-09-14T22:48:22.053 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-09-14T22:48:22.053 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-14T22:48:22.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-14T22:48:22.192 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T22:48:22.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:22.288+0000 7ff72b343640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-14T22:48:22.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:22.289+0000 7ff72b343640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-14T22:48:22.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:22.289+0000 7ff72b343640 -1 bdev(0x55a860e53800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-14T22:48:22.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:22.289+0000 7ff72b343640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-14T22:48:25.198 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-14T22:48:25.229 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T22:48:25.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:25.322+0000 7f458081e640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-14T22:48:25.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:25.322+0000 7f458081e640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-14T22:48:25.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:25.322+0000 7f458081e640 -1 bdev(0x5562f31d3800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-14T22:48:25.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:25.322+0000 7f458081e640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-14T22:48:28.243 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-14T22:48:28.275 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T22:48:28.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:28.366+0000 7f47dae14640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-14T22:48:28.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:28.366+0000 7f47dae14640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-14T22:48:28.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:28.366+0000 7f47dae14640 -1 bdev(0x55bd1ae49800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-14T22:48:28.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-14T22:48:28.366+0000 7f47dae14640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-14T22:48:31.278 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-14T22:48:31.308 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-14T22:48:31.309 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:31.309 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-14T22:48:31.377 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:31.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-14T22:48:31.446 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:31.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-14T22:48:31.514 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:31.514 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-14T22:48:31.582 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:31.582 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-14T22:48:31.640 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:31.640 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-14T22:48:31.659 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:31.660 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-14T22:48:31.717 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:31.718 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-14T22:48:31.774 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:31.775 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-14T22:48:31.831 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:31.831 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-14T22:48:31.888 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:31.888 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-14T22:48:31.945 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:31.945 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-14T22:48:32.002 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:32.002 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-14T22:48:32.058 INFO:tasks.ceph:Adding keys to all mons... 2024-09-14T22:48:32.059 DEBUG:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-14T22:48:32.085 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2024-09-14T22:48:32.085 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC0EuZm2KqjIRAAieoDY5q+iJAU6uJf0atYdQ== 2024-09-14T22:48:32.085 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2024-09-14T22:48:32.085 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC2EuZm/oZHERAAI3/hU28cPW4dUySK806Ylg== 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC5EuZmuV8/ExAA87qAy1fFCyCTGe4xU97rOA== 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC8EuZmK43iFRAAHNkz29igBoTHNrNA92bhWw== 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC0EuZmduRQJBAAtW5vMTMAyRkKFKjP8O3eyA== 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror] 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC0EuZm+ZNTKBAAhwAyswlu//LEJJEtTiexuA== 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.0] 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC0EuZmFa6MKxAAtCEijhFxlEJiMV80z6ok6A== 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.1] 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC0EuZmOxAbMRAAePvxpjw6/mKJL7gFNKP7wQ== 2024-09-14T22:48:32.086 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.2] 2024-09-14T22:48:32.087 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC0EuZms5C9NhAAISuWnItpVHeymfNKSv+OFQ== 2024-09-14T22:48:32.087 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.3] 2024-09-14T22:48:32.087 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC1EuZmhSuuABAA7OAbIAua9xYCldI93hbumg== 2024-09-14T22:48:32.087 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.4] 2024-09-14T22:48:32.087 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC1EuZm8609BhAAV2DBtA1BP0RFHuJYxCYr0A== 2024-09-14T22:48:32.087 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.5] 2024-09-14T22:48:32.087 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC1EuZm5yTaCxAAfcEqVnzp5G27Koqu2Fjiww== 2024-09-14T22:48:32.087 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.6] 2024-09-14T22:48:32.087 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC1EuZmEJN0ERAAIAC2HVjYuKpSJHd704aAVg== 2024-09-14T22:48:32.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T22:48:32.184 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T22:48:32.236 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T22:48:32.288 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T22:48:32.340 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:48:32.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:48:32.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:48:32.506 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:48:32.563 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:48:32.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:48:32.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:48:32.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:48:32.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:48:32.840 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-14T22:48:32.841 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-14T22:48:32.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-14T22:48:32.964 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-14T22:48:32.990 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T22:48:33.046 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-14T22:48:33.046 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-14T22:48:33.046 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-14T22:48:33.089 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-14T22:48:33.090 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-14T22:48:33.090 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-14T22:48:33.091 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-14T22:48:33.093 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-14T22:48:33.093 INFO:tasks.ceph:Setting crush tunables to default 2024-09-14T22:48:33.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-14T22:48:33.269 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-09-14T22:48:33.281 INFO:tasks.ceph:check_enable_crimson: False 2024-09-14T22:48:33.281 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-14T22:48:33.281 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:33.281 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-14T22:48:33.306 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:33.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-14T22:48:33.372 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:33.372 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-14T22:48:33.438 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster cluster1 osd new d0cb2c92-7fae-48c0-9a16-dd219ce47a96 0 2024-09-14T22:48:33.469 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:33.468+0000 7f761db28040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-14T22:48:33.514 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:33.512+0000 7f761db28040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-14T22:48:33.650 INFO:teuthology.orchestra.run.smithi052.stdout:0 2024-09-14T22:48:33.662 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster cluster1 osd new 69ac61b7-3004-4643-b6f8-280f99d135ec 1 2024-09-14T22:48:33.665 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:33.663+0000 7f761db28040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-14T22:48:33.719 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:33.718+0000 7f761db28040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-14T22:48:33.842 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-09-14T22:48:33.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster cluster1 osd new eb017cba-c46f-4e11-8c91-f395956b7643 2 2024-09-14T22:48:33.891 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:33.889+0000 7f761db28040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-14T22:48:34.010 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:34.008+0000 7f761db28040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-14T22:48:34.072 INFO:teuthology.orchestra.run.smithi052.stdout:2 2024-09-14T22:48:34.084 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-14T22:48:34.084 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-14T22:48:34.086 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-14T22:48:34.086 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-14T22:48:34.086 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-14T22:48:34.089 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-14T22:48:34.089 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-14T22:48:34.089 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-14T22:48:34.091 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-14T22:48:34.091 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T22:48:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:34.270 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":5,"fsid":"51bdee54-6793-4abd-9100-af548f45d456","created":"2024-09-14T22:48:33.177127+0000","modified":"2024-09-14T22:48:34.071469+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T22:48:34.071469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d0cb2c92-7fae-48c0-9a16-dd219ce47a96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69ac61b7-3004-4643-b6f8-280f99d135ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb017cba-c46f-4e11-8c91-f395956b7643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:48:34.282 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-14T22:48:34.282 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-14T22:48:34.400 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:34.399+0000 7f761db28040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-14T22:48:34.452 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:34.451+0000 7f761db28040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-14T22:48:34.585 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-14T22:48:34.611 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:34.610+0000 7f761db28040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-14T22:48:34.881 INFO:teuthology.misc.health.smithi052.stderr:2024-09-14T22:48:34.880+0000 7f68e53ba640 0 --2- 172.21.15.52:0/3315307581 >> v2:172.21.15.52:3300/0 conn(0x7f68e00b27b0 0x7f68e00720e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-14T22:48:34.949 INFO:teuthology.misc.health.smithi052.stdout: 2024-09-14T22:48:34.949 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":5,"fsid":"51bdee54-6793-4abd-9100-af548f45d456","created":"2024-09-14T22:48:33.177127+0000","modified":"2024-09-14T22:48:34.071469+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T22:48:34.071469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d0cb2c92-7fae-48c0-9a16-dd219ce47a96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69ac61b7-3004-4643-b6f8-280f99d135ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb017cba-c46f-4e11-8c91-f395956b7643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:48:34.958 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T22:48:35.039 INFO:tasks.ceph.cluster1.osd.2.smithi052.stderr:2024-09-14T22:48:35.037+0000 7f555f83e640 -1 Falling back to public interface 2024-09-14T22:48:35.308 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T22:48:35.306+0000 7f571722f640 -1 Falling back to public interface 2024-09-14T22:48:35.548 INFO:tasks.ceph.cluster1.osd.1.smithi052.stderr:2024-09-14T22:48:35.547+0000 7fb372215640 -1 Falling back to public interface 2024-09-14T22:48:35.623 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:35.622+0000 7f761db28040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-14T22:48:35.679 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:35.677+0000 7f761db28040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-14T22:48:36.066 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.064+0000 7f761db28040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-14T22:48:36.156 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.154+0000 7f761db28040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-14T22:48:36.268 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.266+0000 7f761db28040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-14T22:48:36.317 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.315+0000 7f761db28040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-14T22:48:36.436 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.435+0000 7f761db28040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-14T22:48:36.492 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.490+0000 7f761db28040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-14T22:48:36.539 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.538+0000 7f761db28040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-14T22:48:36.658 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.656+0000 7f761db28040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-14T22:48:36.711 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.708+0000 7f761db28040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-14T22:48:36.764 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.762+0000 7f761db28040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-14T22:48:36.835 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.833+0000 7f761db28040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-14T22:48:36.881 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.880+0000 7f761db28040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-14T22:48:36.928 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:36.926+0000 7f761db28040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-14T22:48:37.613 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:37.612+0000 7f761db28040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-14T22:48:38.673 INFO:tasks.ceph.cluster1.mgr.x.smithi052.stderr:2024-09-14T22:48:38.671+0000 7f75bd9a7640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-14T22:48:38.849 INFO:tasks.ceph.cluster1.osd.2.smithi052.stderr:2024-09-14T22:48:38.847+0000 7f555f83e640 -1 osd.2 0 log_to_monitors true 2024-09-14T22:48:39.104 INFO:tasks.ceph.cluster1.osd.1.smithi052.stderr:2024-09-14T22:48:39.103+0000 7fb372215640 -1 osd.1 0 log_to_monitors true 2024-09-14T22:48:39.379 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T22:48:39.378+0000 7f571722f640 -1 osd.0 0 log_to_monitors true 2024-09-14T22:48:41.262 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-14T22:48:41.725 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T22:48:41.723+0000 7f5712f99640 -1 osd.0 0 waiting for initial osdmap 2024-09-14T22:48:41.747 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T22:48:41.745+0000 7f570a5a5640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T22:48:41.777 INFO:teuthology.misc.health.smithi052.stdout: 2024-09-14T22:48:41.778 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":7,"fsid":"51bdee54-6793-4abd-9100-af548f45d456","created":"2024-09-14T22:48:33.177127+0000","modified":"2024-09-14T22:48:40.672267+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T22:48:34.071469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d0cb2c92-7fae-48c0-9a16-dd219ce47a96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69ac61b7-3004-4643-b6f8-280f99d135ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb017cba-c46f-4e11-8c91-f395956b7643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:48:41.787 INFO:tasks.ceph.cluster1.osd.1.smithi052.stderr:2024-09-14T22:48:41.785+0000 7fb36db26640 -1 osd.1 0 waiting for initial osdmap 2024-09-14T22:48:41.792 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T22:48:41.799 INFO:tasks.ceph.cluster1.osd.2.smithi052.stderr:2024-09-14T22:48:41.796+0000 7f555b7cd640 -1 osd.2 0 waiting for initial osdmap 2024-09-14T22:48:41.802 INFO:tasks.ceph.cluster1.osd.1.smithi052.stderr:2024-09-14T22:48:41.800+0000 7fb365946640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T22:48:41.812 INFO:tasks.ceph.cluster1.osd.2.smithi052.stderr:2024-09-14T22:48:41.810+0000 7f55535ed640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T22:48:48.096 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-14T22:48:48.390 INFO:teuthology.misc.health.smithi052.stdout: 2024-09-14T22:48:48.390 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":12,"fsid":"51bdee54-6793-4abd-9100-af548f45d456","created":"2024-09-14T22:48:33.177127+0000","modified":"2024-09-14T22:48:46.733510+0000","last_up_change":"2024-09-14T22:48:42.725961+0000","last_in_change":"2024-09-14T22:48:34.071469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T22:48:43.862770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d0cb2c92-7fae-48c0-9a16-dd219ce47a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6809","nonce":2942497514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6811","nonce":2942497514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6815","nonce":2942497514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6813","nonce":2942497514}]},"public_addr":"172.21.15.52:6809/2942497514","cluster_addr":"172.21.15.52:6811/2942497514","heartbeat_back_addr":"172.21.15.52:6815/2942497514","heartbeat_front_addr":"172.21.15.52:6813/2942497514","state":["exists","up"]},{"osd":1,"uuid":"69ac61b7-3004-4643-b6f8-280f99d135ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6817","nonce":881972827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6819","nonce":881972827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6823","nonce":881972827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6821","nonce":881972827}]},"public_addr":"172.21.15.52:6817/881972827","cluster_addr":"172.21.15.52:6819/881972827","heartbeat_back_addr":"172.21.15.52:6823/881972827","heartbeat_front_addr":"172.21.15.52:6821/881972827","state":["exists","up"]},{"osd":2,"uuid":"eb017cba-c46f-4e11-8c91-f395956b7643","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6801","nonce":2826339032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6803","nonce":2826339032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6807","nonce":2826339032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6805","nonce":2826339032}]},"public_addr":"172.21.15.52:6801/2826339032","cluster_addr":"172.21.15.52:6803/2826339032","heartbeat_back_addr":"172.21.15.52:6807/2826339032","heartbeat_front_addr":"172.21.15.52:6805/2826339032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:40.378456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:40.126245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:39.874970+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:48:48.401 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-14T22:48:48.401 INFO:tasks.ceph:Creating RBD pool 2024-09-14T22:48:48.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-14T22:48:48.741 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'rbd' created 2024-09-14T22:48:48.755 DEBUG:teuthology.orchestra.run.smithi052:> rbd --cluster cluster1 pool init rbd 2024-09-14T22:48:51.762 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-14T22:48:51.762 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-14T22:48:51.762 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-14T22:48:51.763 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-14T22:48:51.763 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-14T22:48:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:52.125 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":577461670},{"type":"v1","addr":"172.21.15.52:6825","nonce":577461670}]},"active_addr":"172.21.15.52:6825/577461670","active_change":"2024-09-14T22:48:37.614355+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2056361688}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":3740879282}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2322403238}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":949972928}]}]} 2024-09-14T22:48:52.127 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-14T22:48:52.127 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-14T22:48:52.128 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T22:48:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:52.417 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":16,"fsid":"51bdee54-6793-4abd-9100-af548f45d456","created":"2024-09-14T22:48:33.177127+0000","modified":"2024-09-14T22:48:51.749224+0000","last_up_change":"2024-09-14T22:48:42.725961+0000","last_in_change":"2024-09-14T22:48:34.071469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T22:48:43.862770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T22:48:48.717572+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d0cb2c92-7fae-48c0-9a16-dd219ce47a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6809","nonce":2942497514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6811","nonce":2942497514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6815","nonce":2942497514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6813","nonce":2942497514}]},"public_addr":"172.21.15.52:6809/2942497514","cluster_addr":"172.21.15.52:6811/2942497514","heartbeat_back_addr":"172.21.15.52:6815/2942497514","heartbeat_front_addr":"172.21.15.52:6813/2942497514","state":["exists","up"]},{"osd":1,"uuid":"69ac61b7-3004-4643-b6f8-280f99d135ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6817","nonce":881972827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6819","nonce":881972827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6823","nonce":881972827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6821","nonce":881972827}]},"public_addr":"172.21.15.52:6817/881972827","cluster_addr":"172.21.15.52:6819/881972827","heartbeat_back_addr":"172.21.15.52:6823/881972827","heartbeat_front_addr":"172.21.15.52:6821/881972827","state":["exists","up"]},{"osd":2,"uuid":"eb017cba-c46f-4e11-8c91-f395956b7643","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6801","nonce":2826339032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6803","nonce":2826339032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6807","nonce":2826339032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6805","nonce":2826339032}]},"public_addr":"172.21.15.52:6801/2826339032","cluster_addr":"172.21.15.52:6803/2826339032","heartbeat_back_addr":"172.21.15.52:6807/2826339032","heartbeat_front_addr":"172.21.15.52:6805/2826339032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:40.378456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:40.126245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:39.874970+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:48:52.430 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-14T22:48:52.430 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T22:48:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:52.722 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":16,"fsid":"51bdee54-6793-4abd-9100-af548f45d456","created":"2024-09-14T22:48:33.177127+0000","modified":"2024-09-14T22:48:51.749224+0000","last_up_change":"2024-09-14T22:48:42.725961+0000","last_in_change":"2024-09-14T22:48:34.071469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T22:48:43.862770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T22:48:48.717572+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d0cb2c92-7fae-48c0-9a16-dd219ce47a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6809","nonce":2942497514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6811","nonce":2942497514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6815","nonce":2942497514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6813","nonce":2942497514}]},"public_addr":"172.21.15.52:6809/2942497514","cluster_addr":"172.21.15.52:6811/2942497514","heartbeat_back_addr":"172.21.15.52:6815/2942497514","heartbeat_front_addr":"172.21.15.52:6813/2942497514","state":["exists","up"]},{"osd":1,"uuid":"69ac61b7-3004-4643-b6f8-280f99d135ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6817","nonce":881972827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6819","nonce":881972827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6823","nonce":881972827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6821","nonce":881972827}]},"public_addr":"172.21.15.52:6817/881972827","cluster_addr":"172.21.15.52:6819/881972827","heartbeat_back_addr":"172.21.15.52:6823/881972827","heartbeat_front_addr":"172.21.15.52:6821/881972827","state":["exists","up"]},{"osd":2,"uuid":"eb017cba-c46f-4e11-8c91-f395956b7643","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6801","nonce":2826339032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6803","nonce":2826339032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6807","nonce":2826339032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6805","nonce":2826339032}]},"public_addr":"172.21.15.52:6801/2826339032","cluster_addr":"172.21.15.52:6803/2826339032","heartbeat_back_addr":"172.21.15.52:6807/2826339032","heartbeat_front_addr":"172.21.15.52:6805/2826339032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:40.378456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:40.126245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:39.874970+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:48:52.735 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-14T22:48:52.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-14T22:48:52.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-14T22:48:52.879 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2024-09-14T22:48:52.879 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-14T22:48:52.880 INFO:teuthology.orchestra.run.smithi052.stdout:34359738373 2024-09-14T22:48:52.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-14T22:48:52.881 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2024-09-14T22:48:52.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-14T22:48:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2024-09-14T22:48:53.233 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2024-09-14T22:48:53.239 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2024-09-14T22:48:53.247 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2024-09-14T22:48:53.253 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2 2024-09-14T22:48:53.260 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2024-09-14T22:48:54.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-14T22:48:54.254 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-14T22:48:54.261 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-14T22:48:54.575 INFO:teuthology.orchestra.run.smithi052.stdout:34359738373 2024-09-14T22:48:54.584 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2024-09-14T22:48:54.588 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2024-09-14T22:48:54.588 DEBUG:teuthology.parallel:result is None 2024-09-14T22:48:54.597 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.2 2024-09-14T22:48:54.597 DEBUG:teuthology.parallel:result is None 2024-09-14T22:48:54.618 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2024-09-14T22:48:54.631 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0 2024-09-14T22:48:54.631 DEBUG:teuthology.parallel:result is None 2024-09-14T22:48:54.631 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-14T22:48:54.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T22:48:54.961 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:54.961 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-14T22:48:54.973 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-14T22:48:53.620383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94104,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021416,"statfs":{"total":289910292480,"available":289813929984,"internally_reserved":0,"allocated":540672,"data_stored":1303162,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.879812"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.757209+0000","last_change":"2024-09-14T22:48:51.757209+0000","last_active":"2024-09-14T22:48:51.757209+0000","last_peered":"2024-09-14T22:48:51.757209+0000","last_clean":"2024-09-14T22:48:51.757209+0000","last_became_active":"2024-09-14T22:48:49.753482+0000","last_became_peered":"2024-09-14T22:48:49.753482+0000","last_unstale":"2024-09-14T22:48:51.757209+0000","last_undegraded":"2024-09-14T22:48:51.757209+0000","last_fullsized":"2024-09-14T22:48:51.757209+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:13:32.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011426839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.756962+0000","last_change":"2024-09-14T22:48:51.756962+0000","last_active":"2024-09-14T22:48:51.756962+0000","last_peered":"2024-09-14T22:48:51.756962+0000","last_clean":"2024-09-14T22:48:51.756962+0000","last_became_active":"2024-09-14T22:48:49.753595+0000","last_became_peered":"2024-09-14T22:48:49.753595+0000","last_unstale":"2024-09-14T22:48:51.756962+0000","last_undegraded":"2024-09-14T22:48:51.756962+0000","last_fullsized":"2024-09-14T22:48:51.756962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.951246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091631000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.756913+0000","last_change":"2024-09-14T22:48:51.756913+0000","last_active":"2024-09-14T22:48:51.756913+0000","last_peered":"2024-09-14T22:48:51.756913+0000","last_clean":"2024-09-14T22:48:51.756913+0000","last_became_active":"2024-09-14T22:48:49.753420+0000","last_became_peered":"2024-09-14T22:48:49.753420+0000","last_unstale":"2024-09-14T22:48:51.756913+0000","last_undegraded":"2024-09-14T22:48:51.756913+0000","last_fullsized":"2024-09-14T22:48:51.756913+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:38.693996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086764700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.756569+0000","last_change":"2024-09-14T22:48:51.756569+0000","last_active":"2024-09-14T22:48:51.756569+0000","last_peered":"2024-09-14T22:48:51.756569+0000","last_clean":"2024-09-14T22:48:51.756569+0000","last_became_active":"2024-09-14T22:48:49.752095+0000","last_became_peered":"2024-09-14T22:48:49.752095+0000","last_unstale":"2024-09-14T22:48:51.756569+0000","last_undegraded":"2024-09-14T22:48:51.756569+0000","last_fullsized":"2024-09-14T22:48:51.756569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:37:07.776688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054311300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.755421+0000","last_change":"2024-09-14T22:48:51.753263+0000","last_active":"2024-09-14T22:48:51.755421+0000","last_peered":"2024-09-14T22:48:51.755421+0000","last_clean":"2024-09-14T22:48:51.755421+0000","last_became_active":"2024-09-14T22:48:49.752329+0000","last_became_peered":"2024-09-14T22:48:49.752329+0000","last_unstale":"2024-09-14T22:48:51.755421+0000","last_undegraded":"2024-09-14T22:48:51.755421+0000","last_fullsized":"2024-09-14T22:48:51.755421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:34:30.769014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039893099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.803183+0000","last_change":"2024-09-14T22:48:51.803183+0000","last_active":"2024-09-14T22:48:51.803183+0000","last_peered":"2024-09-14T22:48:51.803183+0000","last_clean":"2024-09-14T22:48:51.803183+0000","last_became_active":"2024-09-14T22:48:49.752614+0000","last_became_peered":"2024-09-14T22:48:49.752614+0000","last_unstale":"2024-09-14T22:48:51.803183+0000","last_undegraded":"2024-09-14T22:48:51.803183+0000","last_fullsized":"2024-09-14T22:48:51.803183+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:04:25.567425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045173899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.803192+0000","last_change":"2024-09-14T22:48:51.803192+0000","last_active":"2024-09-14T22:48:51.803192+0000","last_peered":"2024-09-14T22:48:51.803192+0000","last_clean":"2024-09-14T22:48:51.803192+0000","last_became_active":"2024-09-14T22:48:49.753296+0000","last_became_peered":"2024-09-14T22:48:49.753296+0000","last_unstale":"2024-09-14T22:48:51.803192+0000","last_undegraded":"2024-09-14T22:48:51.803192+0000","last_fullsized":"2024-09-14T22:48:51.803192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T22:50:00.939574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045554200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.756500+0000","last_change":"2024-09-14T22:48:51.756500+0000","last_active":"2024-09-14T22:48:51.756500+0000","last_peered":"2024-09-14T22:48:51.756500+0000","last_clean":"2024-09-14T22:48:51.756500+0000","last_became_active":"2024-09-14T22:48:49.752332+0000","last_became_peered":"2024-09-14T22:48:49.752332+0000","last_unstale":"2024-09-14T22:48:51.756500+0000","last_undegraded":"2024-09-14T22:48:51.756500+0000","last_fullsized":"2024-09-14T22:48:51.756500+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:11:52.254597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061883999999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.755748+0000","last_change":"2024-09-14T22:48:46.321941+0000","last_active":"2024-09-14T22:48:51.755748+0000","last_peered":"2024-09-14T22:48:51.755748+0000","last_clean":"2024-09-14T22:48:51.755748+0000","last_became_active":"2024-09-14T22:48:46.320719+0000","last_became_peered":"2024-09-14T22:48:46.320719+0000","last_unstale":"2024-09-14T22:48:51.755748+0000","last_undegraded":"2024-09-14T22:48:51.755748+0000","last_fullsized":"2024-09-14T22:48:51.755748+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_clean_scrub_stamp":"2024-09-14T22:48:44.727263+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:47.941296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":139264,"data_stored":40796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T22:48:54.974 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T22:48:55.258 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:55.259 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-14T22:48:55.270 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-14T22:48:53.620383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94104,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021416,"statfs":{"total":289910292480,"available":289813929984,"internally_reserved":0,"allocated":540672,"data_stored":1303162,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.879812"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.757209+0000","last_change":"2024-09-14T22:48:51.757209+0000","last_active":"2024-09-14T22:48:51.757209+0000","last_peered":"2024-09-14T22:48:51.757209+0000","last_clean":"2024-09-14T22:48:51.757209+0000","last_became_active":"2024-09-14T22:48:49.753482+0000","last_became_peered":"2024-09-14T22:48:49.753482+0000","last_unstale":"2024-09-14T22:48:51.757209+0000","last_undegraded":"2024-09-14T22:48:51.757209+0000","last_fullsized":"2024-09-14T22:48:51.757209+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:13:32.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011426839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.756962+0000","last_change":"2024-09-14T22:48:51.756962+0000","last_active":"2024-09-14T22:48:51.756962+0000","last_peered":"2024-09-14T22:48:51.756962+0000","last_clean":"2024-09-14T22:48:51.756962+0000","last_became_active":"2024-09-14T22:48:49.753595+0000","last_became_peered":"2024-09-14T22:48:49.753595+0000","last_unstale":"2024-09-14T22:48:51.756962+0000","last_undegraded":"2024-09-14T22:48:51.756962+0000","last_fullsized":"2024-09-14T22:48:51.756962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.951246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091631000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.756913+0000","last_change":"2024-09-14T22:48:51.756913+0000","last_active":"2024-09-14T22:48:51.756913+0000","last_peered":"2024-09-14T22:48:51.756913+0000","last_clean":"2024-09-14T22:48:51.756913+0000","last_became_active":"2024-09-14T22:48:49.753420+0000","last_became_peered":"2024-09-14T22:48:49.753420+0000","last_unstale":"2024-09-14T22:48:51.756913+0000","last_undegraded":"2024-09-14T22:48:51.756913+0000","last_fullsized":"2024-09-14T22:48:51.756913+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:38.693996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086764700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.756569+0000","last_change":"2024-09-14T22:48:51.756569+0000","last_active":"2024-09-14T22:48:51.756569+0000","last_peered":"2024-09-14T22:48:51.756569+0000","last_clean":"2024-09-14T22:48:51.756569+0000","last_became_active":"2024-09-14T22:48:49.752095+0000","last_became_peered":"2024-09-14T22:48:49.752095+0000","last_unstale":"2024-09-14T22:48:51.756569+0000","last_undegraded":"2024-09-14T22:48:51.756569+0000","last_fullsized":"2024-09-14T22:48:51.756569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:37:07.776688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054311300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.755421+0000","last_change":"2024-09-14T22:48:51.753263+0000","last_active":"2024-09-14T22:48:51.755421+0000","last_peered":"2024-09-14T22:48:51.755421+0000","last_clean":"2024-09-14T22:48:51.755421+0000","last_became_active":"2024-09-14T22:48:49.752329+0000","last_became_peered":"2024-09-14T22:48:49.752329+0000","last_unstale":"2024-09-14T22:48:51.755421+0000","last_undegraded":"2024-09-14T22:48:51.755421+0000","last_fullsized":"2024-09-14T22:48:51.755421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:34:30.769014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039893099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.803183+0000","last_change":"2024-09-14T22:48:51.803183+0000","last_active":"2024-09-14T22:48:51.803183+0000","last_peered":"2024-09-14T22:48:51.803183+0000","last_clean":"2024-09-14T22:48:51.803183+0000","last_became_active":"2024-09-14T22:48:49.752614+0000","last_became_peered":"2024-09-14T22:48:49.752614+0000","last_unstale":"2024-09-14T22:48:51.803183+0000","last_undegraded":"2024-09-14T22:48:51.803183+0000","last_fullsized":"2024-09-14T22:48:51.803183+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:04:25.567425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045173899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.803192+0000","last_change":"2024-09-14T22:48:51.803192+0000","last_active":"2024-09-14T22:48:51.803192+0000","last_peered":"2024-09-14T22:48:51.803192+0000","last_clean":"2024-09-14T22:48:51.803192+0000","last_became_active":"2024-09-14T22:48:49.753296+0000","last_became_peered":"2024-09-14T22:48:49.753296+0000","last_unstale":"2024-09-14T22:48:51.803192+0000","last_undegraded":"2024-09-14T22:48:51.803192+0000","last_fullsized":"2024-09-14T22:48:51.803192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T22:50:00.939574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045554200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.756500+0000","last_change":"2024-09-14T22:48:51.756500+0000","last_active":"2024-09-14T22:48:51.756500+0000","last_peered":"2024-09-14T22:48:51.756500+0000","last_clean":"2024-09-14T22:48:51.756500+0000","last_became_active":"2024-09-14T22:48:49.752332+0000","last_became_peered":"2024-09-14T22:48:49.752332+0000","last_unstale":"2024-09-14T22:48:51.756500+0000","last_undegraded":"2024-09-14T22:48:51.756500+0000","last_fullsized":"2024-09-14T22:48:51.756500+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:11:52.254597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061883999999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:48:51.755748+0000","last_change":"2024-09-14T22:48:46.321941+0000","last_active":"2024-09-14T22:48:51.755748+0000","last_peered":"2024-09-14T22:48:51.755748+0000","last_clean":"2024-09-14T22:48:51.755748+0000","last_became_active":"2024-09-14T22:48:46.320719+0000","last_became_peered":"2024-09-14T22:48:46.320719+0000","last_unstale":"2024-09-14T22:48:51.755748+0000","last_undegraded":"2024-09-14T22:48:51.755748+0000","last_fullsized":"2024-09-14T22:48:51.755748+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_clean_scrub_stamp":"2024-09-14T22:48:44.727263+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:47.941296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":139264,"data_stored":40796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T22:48:55.271 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-14T22:48:55.271 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-14T22:48:55.271 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-14T22:48:55.272 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-14T22:48:55.593 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:55.593 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T22:48:55.606 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-14T22:48:55.606 INFO:teuthology.run_tasks:Running task ceph... 2024-09-14T22:48:55.615 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-14T22:48:55.615 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-14T22:48:55.616 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7904733', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 4271, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'sleep_before_teardown': 0, 'subset': '97/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'targets': {'smithi052.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPg450sCgqSZFEMi05UZtTnatCArbuCBeU52Co/YZUkC5+nBt0mPDFFs+mOtIM2CF14NE+PwBDTR8HrPFF6sMvg=', 'smithi089.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOphTnuc232wFEHlDEc7DtT8nVVpon/sJ/e1bz9jx+3kNyW0F3b1ZugF+/ty2d3A/yCN1Y97GFGyi2SJGs5K0AM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-14_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-14T22:48:55.616 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-14T22:48:55.618 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-14T22:48:55.637 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T22:48:55.677 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T22:48:55.707 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:55.707 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T22:48:55.766 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T22:48:55.766 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-09-14T22:48:55.824 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T22:48:55.824 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T22:48:55.824 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-14T22:48:55.824 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T22:48:55.824 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T22:48:55.825 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-14 22:48:11.211522203 +0000 2024-09-14T22:48:55.825 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-14 22:45:43.673003849 +0000 2024-09-14T22:48:55.825 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-14 22:45:43.673003849 +0000 2024-09-14T22:48:55.825 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-14 22:45:43.673003849 +0000 2024-09-14T22:48:55.825 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T22:48:55.890 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-14T22:48:55.891 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-14T22:48:55.891 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.00022368 s, 2.3 MB/s 2024-09-14T22:48:55.892 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T22:48:55.951 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-09-14T22:48:56.008 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T22:48:56.008 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T22:48:56.008 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-14T22:48:56.008 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T22:48:56.008 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T22:48:56.008 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-14 22:48:11.211522203 +0000 2024-09-14T22:48:56.008 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-14 22:45:44.006996165 +0000 2024-09-14T22:48:56.009 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-14 22:45:44.006996165 +0000 2024-09-14T22:48:56.009 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-14 22:45:44.006996165 +0000 2024-09-14T22:48:56.009 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T22:48:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-14T22:48:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-14T22:48:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000196314 s, 2.6 MB/s 2024-09-14T22:48:56.078 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T22:48:56.136 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-09-14T22:48:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T22:48:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T22:48:56.193 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-14T22:48:56.193 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T22:48:56.193 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T22:48:56.193 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-14 22:48:11.211522203 +0000 2024-09-14T22:48:56.193 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-14 22:45:44.332988665 +0000 2024-09-14T22:48:56.193 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-14 22:45:44.332988665 +0000 2024-09-14T22:48:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-14 22:45:44.332988665 +0000 2024-09-14T22:48:56.193 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T22:48:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-14T22:48:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-14T22:48:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000178089 s, 2.9 MB/s 2024-09-14T22:48:56.259 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T22:48:56.316 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-09-14T22:48:56.373 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T22:48:56.373 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T22:48:56.373 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-14T22:48:56.373 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T22:48:56.373 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T22:48:56.373 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-14 22:48:11.212522180 +0000 2024-09-14T22:48:56.373 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-14 22:45:44.666980982 +0000 2024-09-14T22:48:56.373 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-14 22:45:44.666980982 +0000 2024-09-14T22:48:56.373 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-14 22:45:44.666980982 +0000 2024-09-14T22:48:56.374 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T22:48:56.436 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-14T22:48:56.437 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-14T22:48:56.437 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000188452 s, 2.7 MB/s 2024-09-14T22:48:56.438 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T22:48:56.494 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T22:48:56.494 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T22:48:56.494 INFO:tasks.ceph:Generating config... 2024-09-14T22:48:56.495 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-14T22:48:56.495 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client] log to stderr = False 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:56.496 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:56.496 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-14T22:48:56.496 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-14T22:48:56.496 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-14T22:48:56.497 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-14T22:48:56.497 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-14T22:48:56.497 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-14T22:48:56.497 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-14T22:48:56.497 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-14T22:48:56.497 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-14T22:48:56.498 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-14T22:48:56.498 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-14T22:48:56.498 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-14T22:48:56.498 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-14T22:48:56.498 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-14T22:48:56.498 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-14T22:48:56.575 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster2.keyring 2024-09-14T22:48:56.578 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-14T22:48:56.661 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-14T22:48:56.726 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.89')] 2024-09-14T22:48:56.726 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.89', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-14T22:48:56.727 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:56.727 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T22:48:56.781 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.89 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T22:48:56.861 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T22:48:56.861 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: generated fsid 666df62e-9042-4f73-97d4-b9e02ade1b08 2024-09-14T22:48:56.862 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = quincy 2024-09-14T22:48:56.862 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 0 2024-09-14T22:48:56.862 INFO:teuthology.orchestra.run.smithi089.stdout:fsid 666df62e-9042-4f73-97d4-b9e02ade1b08 2024-09-14T22:48:56.862 INFO:teuthology.orchestra.run.smithi089.stdout:last_changed 2024-09-14T22:48:56.860702+0000 2024-09-14T22:48:56.862 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-09-14T22:48:56.860702+0000 2024-09-14T22:48:56.862 INFO:teuthology.orchestra.run.smithi089.stdout:min_mon_release 17 (quincy) 2024-09-14T22:48:56.863 INFO:teuthology.orchestra.run.smithi089.stdout:election_strategy: 1 2024-09-14T22:48:56.863 INFO:teuthology.orchestra.run.smithi089.stdout:0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-14T22:48:56.863 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-14T22:48:56.864 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T22:48:56.919 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 666df62e-9042-4f73-97d4-b9e02ade1b08... 2024-09-14T22:48:56.920 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-14T22:48:56.923 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-14T22:48:56.973 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-09-14T22:48:56.973 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2024-09-14T22:48:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = 172.21.15.89 2024-09-14T22:48:56.976 INFO:teuthology.orchestra.run.smithi052.stdout: mon client directed command retry = 5 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on pool no app = False 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: ms inject socket failures = 5000 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 666df62e-9042-4f73-97d4-b9e02ade1b08 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock profile = high_recovery_ops 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2024-09-14T22:48:56.977 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug trim objects = true 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2024-09-14T22:48:56.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore block size = 96636764160 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore compression algorithm = zstd 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore compression mode = aggressive 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore fsck on mount = True 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluefs = 1/20 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluestore = 1/20 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 20 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: debug rocksdb = 4/10 2024-09-14T22:48:56.979 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd full ratio = 0.9 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd nearfull ratio = 0.8 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: osd failsafe full ratio = 0.95 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: osd objectstore = bluestore 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.980 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2024-09-14T22:48:56.981 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-09-14T22:48:56.981 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-09-14T22:48:56.981 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2024-09-14T22:48:56.981 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-09-14T22:48:56.981 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2024-09-14T22:48:56.981 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2024-09-14T22:48:56.981 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-09-14T22:48:56.981 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: # 1m isn't quite enough 2024-09-14T22:48:56.982 INFO:teuthology.orchestra.run.smithi052.stdout: mon_down_mkfs_grace = 2m 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_filestore_osds = false 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: debug rbd = 20 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: debug rbd_mirror = 15 2024-09-14T22:48:56.983 INFO:teuthology.orchestra.run.smithi052.stdout: log to stderr = False 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout: rbd default features = 125 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout: rbd mirror image policy type = simple 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.0] 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.1] 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.2] 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout:[client.mirror.3] 2024-09-14T22:48:56.984 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:56.985 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-09-14T22:48:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T22:48:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = 172.21.15.89 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: mon client directed command retry = 5 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on pool no app = False 2024-09-14T22:48:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject socket failures = 5000 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 666df62e-9042-4f73-97d4-b9e02ade1b08 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-09-14T22:48:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug trim objects = true 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-09-14T22:48:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore compression algorithm = zstd 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore compression mode = aggressive 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-09-14T22:48:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-09-14T22:48:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T22:48:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: debug rbd = 20 2024-09-14T22:48:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: debug rbd_mirror = 15 2024-09-14T22:48:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: log to stderr = False 2024-09-14T22:48:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: rbd default features = 125 2024-09-14T22:48:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: rbd mirror image policy type = simple 2024-09-14T22:48:57.013 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-09-14T22:48:57.013 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.0] 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.1] 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.2] 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.3] 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T22:48:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T22:48:57.015 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-14T22:48:57.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-14T22:48:57.110 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-14T22:48:57.111 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:57.111 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-14T22:48:57.126 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:57.127 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-14T22:48:57.181 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T22:48:57.181 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:57.182 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-14T22:48:57.182 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-14T22:48:57.223 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:48:57.224 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-14T22:48:57.281 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T22:48:57.281 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:57.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-14T22:48:57.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-14T22:48:57.316 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:48:57.317 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-14T22:48:57.372 INFO:tasks.ceph:Setting up mon nodes... 2024-09-14T22:48:57.372 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-14T22:48:57.373 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-14T22:48:57.430 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-14T22:48:57.433 INFO:tasks.ceph:Setting up mds nodes... 2024-09-14T22:48:57.433 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-14T22:48:57.434 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-14T22:48:57.477 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-14T22:48:57.491 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-14T22:48:57.533 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-14T22:48:57.546 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-14T22:48:57.627 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-14T22:48:57.640 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-14T22:48:57.722 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-14T22:48:57.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-14T22:48:57.814 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-14T22:48:57.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-14T22:48:57.908 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-14T22:48:57.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-14T22:48:58.002 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-14T22:48:58.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-14T22:48:58.097 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-14T22:48:58.109 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-14T22:48:58.190 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-14T22:48:58.203 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-14T22:48:58.203 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T22:48:58.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-14T22:48:58.268 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T22:48:58.268 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-14T22:48:58.268 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T22:48:58.269 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-14T22:48:58.343 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T22:48:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T22:48:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T22:48:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T22:48:58.343 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T22:48:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-09-14T22:48:58.344 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T22:48:58.344 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T22:48:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T22:48:58.344 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T22:48:58.824 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-09-14T22:48:58.826 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-09-14T22:48:58.826 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-14T22:48:58.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-14T22:48:59.002 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-14T22:48:59.069 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T22:48:59.069 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-14T22:48:59.069 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T22:48:59.070 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T22:48:59.143 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T22:48:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-09-14T22:48:59.606 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-09-14T22:48:59.606 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-14T22:48:59.695 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-14T22:48:59.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-14T22:48:59.791 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T22:48:59.791 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-14T22:48:59.791 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T22:48:59.791 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-14T22:48:59.864 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T22:48:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T22:48:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T22:48:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T22:48:59.864 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T22:48:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-09-14T22:48:59.865 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T22:48:59.865 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T22:48:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T22:48:59.865 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T22:49:00.354 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-09-14T22:49:00.356 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-09-14T22:49:00.357 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-14T22:49:00.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-14T22:49:00.504 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T22:49:00.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:00.608+0000 7fba18db2640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-14T22:49:00.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:00.609+0000 7fba18db2640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-14T22:49:00.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:00.609+0000 7fba18db2640 -1 bdev(0x5588caecf800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-14T22:49:00.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:00.610+0000 7fba18db2640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-14T22:49:03.516 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-14T22:49:03.546 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T22:49:03.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:03.636+0000 7feeaa9b0640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-14T22:49:03.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:03.636+0000 7feeaa9b0640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-14T22:49:03.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:03.636+0000 7feeaa9b0640 -1 bdev(0x55d05c58b800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-14T22:49:03.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:03.636+0000 7feeaa9b0640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-14T22:49:06.548 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-14T22:49:06.578 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T22:49:06.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:06.667+0000 7fb8891b9640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-14T22:49:06.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:06.667+0000 7fb8891b9640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-14T22:49:06.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:06.667+0000 7fb8891b9640 -1 bdev(0x5571cf68d800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-14T22:49:06.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-14T22:49:06.668+0000 7fb8891b9640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-14T22:49:09.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-14T22:49:09.606 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-14T22:49:09.606 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:49:09.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-14T22:49:09.638 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:09.638 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-14T22:49:09.673 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:09.673 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-14T22:49:09.739 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:09.740 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-14T22:49:09.805 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-14T22:49:09.805 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-14T22:49:09.825 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:09.825 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-14T22:49:09.861 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:09.861 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-14T22:49:09.917 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:09.917 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-14T22:49:09.973 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:09.973 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-14T22:49:10.029 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:10.029 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-14T22:49:10.084 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:10.085 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-14T22:49:10.139 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:10.140 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-14T22:49:10.194 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:10.195 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-14T22:49:10.249 INFO:tasks.ceph:Adding keys to all mons... 2024-09-14T22:49:10.249 DEBUG:teuthology.orchestra.run.smithi089:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-14T22:49:10.312 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.x] 2024-09-14T22:49:10.312 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZm0YGiGRAA6oXI6+nRm5NOMRwumCLDiA== 2024-09-14T22:49:10.312 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.0] 2024-09-14T22:49:10.312 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDcEuZmM8NMJBAAVBU8QESJLh54l6R7fYrwqQ== 2024-09-14T22:49:10.312 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.1] 2024-09-14T22:49:10.312 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDfEuZmz5b7JRAA2QfDXQlcCwf36E/krBBHEQ== 2024-09-14T22:49:10.312 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.2] 2024-09-14T22:49:10.312 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDiEuZmmKzWJxAAOZbeTjiSdQ880moALwJSJQ== 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZmem5vHBAASO9jxFV8946DX3e++Vv+oA== 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror] 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZmWei+HxAAvwVOUS2eJ45Yly5d77tAFQ== 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.0] 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZmspRcJRAABkTuYS1j6xwviWPjALAnZg== 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.1] 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZmEXMBKxAAIepEg/ldJnxCuXOtjCwA/g== 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.2] 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZm292LMBAAJzRJDjw3ZWhG4QZTpAxM2Q== 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.3] 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZm+9obNhAAzMU064b03XQ0jvmG5XMPlA== 2024-09-14T22:49:10.313 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.4] 2024-09-14T22:49:10.314 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDaEuZm+J0cABAAwgfPyhRPz31SKMtXrAyE7g== 2024-09-14T22:49:10.314 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.5] 2024-09-14T22:49:10.314 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDaEuZmCcPBBRAAPgPnLrfPbuNu3d4S5En/HQ== 2024-09-14T22:49:10.314 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.6] 2024-09-14T22:49:10.314 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDaEuZmNQJQCxAA8xtBW/gqyJqu8xYJrxSk8Q== 2024-09-14T22:49:10.315 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T22:49:10.406 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T22:49:10.458 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T22:49:10.508 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T22:49:10.559 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:49:10.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:49:10.665 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:49:10.718 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:49:10.771 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:49:10.824 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:49:10.878 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:49:10.933 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:49:10.986 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T22:49:11.040 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-14T22:49:11.040 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-14T22:49:11.064 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-14T22:49:11.159 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-14T22:49:11.184 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T22:49:11.240 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-14T22:49:11.240 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-14T22:49:11.240 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-14T22:49:11.283 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-14T22:49:11.283 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-14T22:49:11.283 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-14T22:49:11.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-14T22:49:11.286 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-14T22:49:11.286 INFO:tasks.ceph:Setting crush tunables to default 2024-09-14T22:49:11.287 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-14T22:49:11.453 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2024-09-14T22:49:11.458 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:11.457+0000 7fefcab97040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-14T22:49:11.465 INFO:tasks.ceph:check_enable_crimson: False 2024-09-14T22:49:11.465 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-14T22:49:11.465 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:11.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-14T22:49:11.489 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:11.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-14T22:49:11.501 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:11.499+0000 7fefcab97040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-14T22:49:11.554 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:49:11.554 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-14T22:49:11.618 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster cluster2 osd new a9565fd8-ae8a-4ebe-af3b-2c4cac8f4eeb 0 2024-09-14T22:49:11.637 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:11.636+0000 7fefcab97040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-14T22:49:11.687 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:11.686+0000 7fefcab97040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-14T22:49:11.800 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:11.798+0000 7fefcab97040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-14T22:49:11.829 INFO:teuthology.orchestra.run.smithi089.stdout:0 2024-09-14T22:49:11.840 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster cluster2 osd new 604fb917-9fee-4429-82c2-e8f45cc16900 1 2024-09-14T22:49:11.896 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:11.895+0000 7fefcab97040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-14T22:49:12.012 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-09-14T22:49:12.023 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster cluster2 osd new 670d215b-6b09-4e95-a9eb-f18c22b78f7d 2 2024-09-14T22:49:12.169 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:12.167+0000 7fefcab97040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-14T22:49:12.198 INFO:teuthology.orchestra.run.smithi089.stdout:2 2024-09-14T22:49:12.209 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-14T22:49:12.209 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-14T22:49:12.211 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-14T22:49:12.211 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-14T22:49:12.211 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-14T22:49:12.213 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-14T22:49:12.213 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-14T22:49:12.213 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-14T22:49:12.215 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-14T22:49:12.215 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T22:49:12.216 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:12.213+0000 7fefcab97040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-14T22:49:12.336 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:12.334+0000 7fefcab97040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-14T22:49:12.381 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:49:12.382 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":5,"fsid":"666df62e-9042-4f73-97d4-b9e02ade1b08","created":"2024-09-14T22:49:11.368946+0000","modified":"2024-09-14T22:49:12.196753+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T22:49:12.196753+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a9565fd8-ae8a-4ebe-af3b-2c4cac8f4eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"604fb917-9fee-4429-82c2-e8f45cc16900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"670d215b-6b09-4e95-a9eb-f18c22b78f7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:49:12.392 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-14T22:49:12.393 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-14T22:49:12.695 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-14T22:49:12.851 INFO:teuthology.misc.health.smithi089.stdout: 2024-09-14T22:49:12.851 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":5,"fsid":"666df62e-9042-4f73-97d4-b9e02ade1b08","created":"2024-09-14T22:49:11.368946+0000","modified":"2024-09-14T22:49:12.196753+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T22:49:12.196753+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a9565fd8-ae8a-4ebe-af3b-2c4cac8f4eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"604fb917-9fee-4429-82c2-e8f45cc16900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"670d215b-6b09-4e95-a9eb-f18c22b78f7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:49:12.860 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T22:49:13.061 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:13.060+0000 7fefcab97040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-14T22:49:13.107 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:13.105+0000 7fefcab97040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-14T22:49:13.433 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:13.431+0000 7fefcab97040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-14T22:49:13.519 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:13.517+0000 7fefcab97040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-14T22:49:13.630 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:13.629+0000 7fefcab97040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-14T22:49:13.674 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:13.673+0000 7fefcab97040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-14T22:49:13.791 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:13.789+0000 7fefcab97040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-14T22:49:13.844 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:13.842+0000 7fefcab97040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-14T22:49:13.887 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:13.885+0000 7fefcab97040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-14T22:49:13.900 INFO:tasks.ceph.cluster2.osd.0.smithi089.stderr:2024-09-14T22:49:13.897+0000 7fc77980c640 -1 Falling back to public interface 2024-09-14T22:49:13.904 INFO:tasks.ceph.cluster2.osd.1.smithi089.stderr:2024-09-14T22:49:13.902+0000 7fdac3222640 -1 Falling back to public interface 2024-09-14T22:49:14.003 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:14.002+0000 7fefcab97040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-14T22:49:14.053 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:14.052+0000 7fefcab97040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-14T22:49:14.104 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:14.101+0000 7fefcab97040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-14T22:49:14.125 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T22:49:14.123+0000 7f5ff1812640 -1 Falling back to public interface 2024-09-14T22:49:14.157 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:14.155+0000 7fefcab97040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-14T22:49:14.200 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:14.199+0000 7fefcab97040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-14T22:49:14.244 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:14.242+0000 7fefcab97040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-14T22:49:14.829 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:14.828+0000 7fefcab97040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-14T22:49:16.098 INFO:tasks.ceph.cluster2.mgr.x.smithi052.stderr:2024-09-14T22:49:16.096+0000 7fef6b4a7640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-14T22:49:17.452 INFO:tasks.ceph.cluster2.osd.1.smithi089.stderr:2024-09-14T22:49:17.451+0000 7fdac3222640 -1 osd.1 0 log_to_monitors true 2024-09-14T22:49:17.912 INFO:tasks.ceph.cluster2.osd.0.smithi089.stderr:2024-09-14T22:49:17.910+0000 7fc77980c640 -1 osd.0 0 log_to_monitors true 2024-09-14T22:49:18.369 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T22:49:18.367+0000 7f5ff1812640 -1 osd.2 0 log_to_monitors true 2024-09-14T22:49:19.164 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-14T22:49:19.499 INFO:teuthology.misc.health.smithi089.stdout: 2024-09-14T22:49:19.500 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":7,"fsid":"666df62e-9042-4f73-97d4-b9e02ade1b08","created":"2024-09-14T22:49:11.368946+0000","modified":"2024-09-14T22:49:18.896719+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T22:49:12.196753+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a9565fd8-ae8a-4ebe-af3b-2c4cac8f4eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"604fb917-9fee-4429-82c2-e8f45cc16900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"670d215b-6b09-4e95-a9eb-f18c22b78f7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:49:19.512 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T22:49:19.648 INFO:tasks.ceph.cluster2.osd.1.smithi089.stderr:2024-09-14T22:49:19.646+0000 7fdabf328640 -1 osd.1 0 waiting for initial osdmap 2024-09-14T22:49:19.658 INFO:tasks.ceph.cluster2.osd.1.smithi089.stderr:2024-09-14T22:49:19.656+0000 7fdab6934640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T22:49:20.755 INFO:tasks.ceph.cluster2.osd.0.smithi089.stderr:2024-09-14T22:49:20.753+0000 7fc7750c9640 -1 osd.0 0 waiting for initial osdmap 2024-09-14T22:49:20.765 INFO:tasks.ceph.cluster2.osd.0.smithi089.stderr:2024-09-14T22:49:20.763+0000 7fc76cee9640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T22:49:20.768 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T22:49:20.766+0000 7f5fed1e0640 -1 osd.2 0 waiting for initial osdmap 2024-09-14T22:49:20.781 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T22:49:20.779+0000 7f5fe5000640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T22:49:25.815 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-14T22:49:26.095 INFO:teuthology.misc.health.smithi089.stdout: 2024-09-14T22:49:26.095 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":12,"fsid":"666df62e-9042-4f73-97d4-b9e02ade1b08","created":"2024-09-14T22:49:11.368946+0000","modified":"2024-09-14T22:49:23.915213+0000","last_up_change":"2024-09-14T22:49:20.911648+0000","last_in_change":"2024-09-14T22:49:12.196753+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T22:49:21.064218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9565fd8-ae8a-4ebe-af3b-2c4cac8f4eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6801","nonce":1763187606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6803","nonce":1763187606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6807","nonce":1763187606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6805","nonce":1763187606}]},"public_addr":"172.21.15.89:6801/1763187606","cluster_addr":"172.21.15.89:6803/1763187606","heartbeat_back_addr":"172.21.15.89:6807/1763187606","heartbeat_front_addr":"172.21.15.89:6805/1763187606","state":["exists","up"]},{"osd":1,"uuid":"604fb917-9fee-4429-82c2-e8f45cc16900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6809","nonce":2899321748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6811","nonce":2899321748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6815","nonce":2899321748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6813","nonce":2899321748}]},"public_addr":"172.21.15.89:6809/2899321748","cluster_addr":"172.21.15.89:6811/2899321748","heartbeat_back_addr":"172.21.15.89:6815/2899321748","heartbeat_front_addr":"172.21.15.89:6813/2899321748","state":["exists","up"]},{"osd":2,"uuid":"670d215b-6b09-4e95-a9eb-f18c22b78f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6817","nonce":35213865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6819","nonce":35213865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6823","nonce":35213865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6821","nonce":35213865}]},"public_addr":"172.21.15.89:6817/35213865","cluster_addr":"172.21.15.89:6819/35213865","heartbeat_back_addr":"172.21.15.89:6823/35213865","heartbeat_front_addr":"172.21.15.89:6821/35213865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:18.864288+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:18.455058+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:19.337726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:49:26.106 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-14T22:49:26.106 INFO:tasks.ceph:Creating RBD pool 2024-09-14T22:49:26.107 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-14T22:49:26.928 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'rbd' created 2024-09-14T22:49:26.943 DEBUG:teuthology.orchestra.run.smithi089:> rbd --cluster cluster2 pool init rbd 2024-09-14T22:49:29.959 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-14T22:49:29.960 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-14T22:49:29.960 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-14T22:49:29.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-14T22:49:29.960 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-14T22:49:30.292 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:49:30.303 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":4168093985},{"type":"v1","addr":"172.21.15.52:6827","nonce":4168093985}]},"active_addr":"172.21.15.52:6827/4168093985","active_change":"2024-09-14T22:49:14.830308+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2812274015}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2065650167}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1054519329}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1263163303}]}]} 2024-09-14T22:49:30.305 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-14T22:49:30.305 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-14T22:49:30.305 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T22:49:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:49:30.580 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":16,"fsid":"666df62e-9042-4f73-97d4-b9e02ade1b08","created":"2024-09-14T22:49:11.368946+0000","modified":"2024-09-14T22:49:29.942159+0000","last_up_change":"2024-09-14T22:49:20.911648+0000","last_in_change":"2024-09-14T22:49:12.196753+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T22:49:21.064218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T22:49:26.396017+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9565fd8-ae8a-4ebe-af3b-2c4cac8f4eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6801","nonce":1763187606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6803","nonce":1763187606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6807","nonce":1763187606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6805","nonce":1763187606}]},"public_addr":"172.21.15.89:6801/1763187606","cluster_addr":"172.21.15.89:6803/1763187606","heartbeat_back_addr":"172.21.15.89:6807/1763187606","heartbeat_front_addr":"172.21.15.89:6805/1763187606","state":["exists","up"]},{"osd":1,"uuid":"604fb917-9fee-4429-82c2-e8f45cc16900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6809","nonce":2899321748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6811","nonce":2899321748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6815","nonce":2899321748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6813","nonce":2899321748}]},"public_addr":"172.21.15.89:6809/2899321748","cluster_addr":"172.21.15.89:6811/2899321748","heartbeat_back_addr":"172.21.15.89:6815/2899321748","heartbeat_front_addr":"172.21.15.89:6813/2899321748","state":["exists","up"]},{"osd":2,"uuid":"670d215b-6b09-4e95-a9eb-f18c22b78f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6817","nonce":35213865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6819","nonce":35213865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6823","nonce":35213865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6821","nonce":35213865}]},"public_addr":"172.21.15.89:6817/35213865","cluster_addr":"172.21.15.89:6819/35213865","heartbeat_back_addr":"172.21.15.89:6823/35213865","heartbeat_front_addr":"172.21.15.89:6821/35213865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:18.864288+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:18.455058+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:19.337726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:49:30.592 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-14T22:49:30.592 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T22:49:30.866 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:49:30.866 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":16,"fsid":"666df62e-9042-4f73-97d4-b9e02ade1b08","created":"2024-09-14T22:49:11.368946+0000","modified":"2024-09-14T22:49:29.942159+0000","last_up_change":"2024-09-14T22:49:20.911648+0000","last_in_change":"2024-09-14T22:49:12.196753+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T22:49:21.064218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T22:49:26.396017+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9565fd8-ae8a-4ebe-af3b-2c4cac8f4eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6801","nonce":1763187606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6803","nonce":1763187606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6807","nonce":1763187606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6805","nonce":1763187606}]},"public_addr":"172.21.15.89:6801/1763187606","cluster_addr":"172.21.15.89:6803/1763187606","heartbeat_back_addr":"172.21.15.89:6807/1763187606","heartbeat_front_addr":"172.21.15.89:6805/1763187606","state":["exists","up"]},{"osd":1,"uuid":"604fb917-9fee-4429-82c2-e8f45cc16900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6809","nonce":2899321748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6811","nonce":2899321748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6815","nonce":2899321748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6813","nonce":2899321748}]},"public_addr":"172.21.15.89:6809/2899321748","cluster_addr":"172.21.15.89:6811/2899321748","heartbeat_back_addr":"172.21.15.89:6815/2899321748","heartbeat_front_addr":"172.21.15.89:6813/2899321748","state":["exists","up"]},{"osd":2,"uuid":"670d215b-6b09-4e95-a9eb-f18c22b78f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6817","nonce":35213865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6819","nonce":35213865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6823","nonce":35213865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6821","nonce":35213865}]},"public_addr":"172.21.15.89:6817/35213865","cluster_addr":"172.21.15.89:6819/35213865","heartbeat_back_addr":"172.21.15.89:6823/35213865","heartbeat_front_addr":"172.21.15.89:6821/35213865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:18.864288+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:18.455058+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:19.337726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T22:49:30.878 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-14T22:49:30.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-14T22:49:30.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-14T22:49:31.016 INFO:teuthology.orchestra.run.smithi089.stdout:38654705669 2024-09-14T22:49:31.016 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-14T22:49:31.032 INFO:teuthology.orchestra.run.smithi089.stdout:34359738373 2024-09-14T22:49:31.032 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-14T22:49:31.033 INFO:teuthology.orchestra.run.smithi089.stdout:38654705668 2024-09-14T22:49:31.033 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-14T22:49:31.308 INFO:teuthology.orchestra.run.smithi089.stdout:38654705668 2024-09-14T22:49:31.321 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.0 2024-09-14T22:49:31.360 INFO:teuthology.orchestra.run.smithi089.stdout:34359738372 2024-09-14T22:49:31.365 INFO:teuthology.orchestra.run.smithi089.stdout:38654705667 2024-09-14T22:49:31.373 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-09-14T22:49:31.378 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705667 for osd.2 2024-09-14T22:49:32.322 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-14T22:49:32.374 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-14T22:49:32.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-14T22:49:32.627 INFO:teuthology.orchestra.run.smithi089.stdout:38654705668 2024-09-14T22:49:32.640 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.0 2024-09-14T22:49:32.668 INFO:teuthology.orchestra.run.smithi089.stdout:34359738372 2024-09-14T22:49:32.681 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-09-14T22:49:32.707 INFO:teuthology.orchestra.run.smithi089.stdout:38654705667 2024-09-14T22:49:32.719 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705667 for osd.2 2024-09-14T22:49:33.641 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-14T22:49:33.683 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-14T22:49:33.720 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-14T22:49:33.946 INFO:teuthology.orchestra.run.smithi089.stdout:38654705670 2024-09-14T22:49:33.959 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705670 for osd.0 2024-09-14T22:49:33.959 DEBUG:teuthology.parallel:result is None 2024-09-14T22:49:33.985 INFO:teuthology.orchestra.run.smithi089.stdout:34359738373 2024-09-14T22:49:33.998 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.1 2024-09-14T22:49:33.998 DEBUG:teuthology.parallel:result is None 2024-09-14T22:49:34.023 INFO:teuthology.orchestra.run.smithi089.stdout:38654705669 2024-09-14T22:49:34.035 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705669 for osd.2 2024-09-14T22:49:34.035 DEBUG:teuthology.parallel:result is None 2024-09-14T22:49:34.035 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-14T22:49:34.036 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T22:49:34.347 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:49:34.347 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-14T22:49:34.359 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-14T22:49:32.837649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":85772,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283029748,"statfs":{"total":289910292480,"available":289822461952,"internally_reserved":0,"allocated":540672,"data_stored":1302991,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":87223633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.906908"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.945195+0000","last_change":"2024-09-14T22:49:29.945463+0000","last_active":"2024-09-14T22:49:29.945195+0000","last_peered":"2024-09-14T22:49:29.945195+0000","last_clean":"2024-09-14T22:49:29.945195+0000","last_became_active":"2024-09-14T22:49:27.947764+0000","last_became_peered":"2024-09-14T22:49:27.947764+0000","last_unstale":"2024-09-14T22:49:29.945195+0000","last_undegraded":"2024-09-14T22:49:29.945195+0000","last_fullsized":"2024-09-14T22:49:29.945195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:23:56.112794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096834600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944926+0000","last_change":"2024-09-14T22:49:29.945152+0000","last_active":"2024-09-14T22:49:29.944926+0000","last_peered":"2024-09-14T22:49:29.944926+0000","last_clean":"2024-09-14T22:49:29.944926+0000","last_became_active":"2024-09-14T22:49:27.948873+0000","last_became_peered":"2024-09-14T22:49:27.948873+0000","last_unstale":"2024-09-14T22:49:29.944926+0000","last_undegraded":"2024-09-14T22:49:29.944926+0000","last_fullsized":"2024-09-14T22:49:29.944926+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:59.551999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069214500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944861+0000","last_change":"2024-09-14T22:49:29.945105+0000","last_active":"2024-09-14T22:49:29.944861+0000","last_peered":"2024-09-14T22:49:29.944861+0000","last_clean":"2024-09-14T22:49:29.944861+0000","last_became_active":"2024-09-14T22:49:27.949153+0000","last_became_peered":"2024-09-14T22:49:27.949153+0000","last_unstale":"2024-09-14T22:49:29.944861+0000","last_undegraded":"2024-09-14T22:49:29.944861+0000","last_fullsized":"2024-09-14T22:49:29.944861+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:58:19.694736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000710178,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944562+0000","last_change":"2024-09-14T22:49:29.944792+0000","last_active":"2024-09-14T22:49:29.944562+0000","last_peered":"2024-09-14T22:49:29.944562+0000","last_clean":"2024-09-14T22:49:29.944562+0000","last_became_active":"2024-09-14T22:49:27.947823+0000","last_became_peered":"2024-09-14T22:49:27.947823+0000","last_unstale":"2024-09-14T22:49:29.944562+0000","last_undegraded":"2024-09-14T22:49:29.944562+0000","last_fullsized":"2024-09-14T22:49:29.944562+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:16:27.031276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043000900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.953578+0000","last_change":"2024-09-14T22:49:29.949279+0000","last_active":"2024-09-14T22:49:29.953578+0000","last_peered":"2024-09-14T22:49:29.953578+0000","last_clean":"2024-09-14T22:49:29.953578+0000","last_became_active":"2024-09-14T22:49:27.948120+0000","last_became_peered":"2024-09-14T22:49:27.948120+0000","last_unstale":"2024-09-14T22:49:29.953578+0000","last_undegraded":"2024-09-14T22:49:29.953578+0000","last_fullsized":"2024-09-14T22:49:29.953578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:53:56.777188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095602899999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.973218+0000","last_change":"2024-09-14T22:49:29.973218+0000","last_active":"2024-09-14T22:49:29.973218+0000","last_peered":"2024-09-14T22:49:29.973218+0000","last_clean":"2024-09-14T22:49:29.973218+0000","last_became_active":"2024-09-14T22:49:27.948225+0000","last_became_peered":"2024-09-14T22:49:27.948225+0000","last_unstale":"2024-09-14T22:49:29.973218+0000","last_undegraded":"2024-09-14T22:49:29.973218+0000","last_fullsized":"2024-09-14T22:49:29.973218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:34.949309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044858000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.973217+0000","last_change":"2024-09-14T22:49:29.973217+0000","last_active":"2024-09-14T22:49:29.973217+0000","last_peered":"2024-09-14T22:49:29.973217+0000","last_clean":"2024-09-14T22:49:29.973217+0000","last_became_active":"2024-09-14T22:49:27.947929+0000","last_became_peered":"2024-09-14T22:49:27.947929+0000","last_unstale":"2024-09-14T22:49:29.973217+0000","last_undegraded":"2024-09-14T22:49:29.973217+0000","last_fullsized":"2024-09-14T22:49:29.973217+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:21:03.176471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944599+0000","last_change":"2024-09-14T22:49:29.944800+0000","last_active":"2024-09-14T22:49:29.944599+0000","last_peered":"2024-09-14T22:49:29.944599+0000","last_clean":"2024-09-14T22:49:29.944599+0000","last_became_active":"2024-09-14T22:49:27.947559+0000","last_became_peered":"2024-09-14T22:49:27.947559+0000","last_unstale":"2024-09-14T22:49:29.944599+0000","last_undegraded":"2024-09-14T22:49:29.944599+0000","last_fullsized":"2024-09-14T22:49:29.944599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:33:43.576216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044691000000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944243+0000","last_change":"2024-09-14T22:49:22.921000+0000","last_active":"2024-09-14T22:49:29.944243+0000","last_peered":"2024-09-14T22:49:29.944243+0000","last_clean":"2024-09-14T22:49:29.944243+0000","last_became_active":"2024-09-14T22:49:22.919993+0000","last_became_peered":"2024-09-14T22:49:22.919993+0000","last_unstale":"2024-09-14T22:49:29.944243+0000","last_undegraded":"2024-09-14T22:49:29.944243+0000","last_fullsized":"2024-09-14T22:49:29.944243+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_clean_scrub_stamp":"2024-09-14T22:49:21.912115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:51.033680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":40739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":631126,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":200704,"data_stored":631126,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T22:49:34.360 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T22:49:34.634 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:49:34.634 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-14T22:49:34.646 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-14T22:49:32.837649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":85772,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283029748,"statfs":{"total":289910292480,"available":289822461952,"internally_reserved":0,"allocated":540672,"data_stored":1302991,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":87223633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.906908"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.945195+0000","last_change":"2024-09-14T22:49:29.945463+0000","last_active":"2024-09-14T22:49:29.945195+0000","last_peered":"2024-09-14T22:49:29.945195+0000","last_clean":"2024-09-14T22:49:29.945195+0000","last_became_active":"2024-09-14T22:49:27.947764+0000","last_became_peered":"2024-09-14T22:49:27.947764+0000","last_unstale":"2024-09-14T22:49:29.945195+0000","last_undegraded":"2024-09-14T22:49:29.945195+0000","last_fullsized":"2024-09-14T22:49:29.945195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:23:56.112794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096834600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944926+0000","last_change":"2024-09-14T22:49:29.945152+0000","last_active":"2024-09-14T22:49:29.944926+0000","last_peered":"2024-09-14T22:49:29.944926+0000","last_clean":"2024-09-14T22:49:29.944926+0000","last_became_active":"2024-09-14T22:49:27.948873+0000","last_became_peered":"2024-09-14T22:49:27.948873+0000","last_unstale":"2024-09-14T22:49:29.944926+0000","last_undegraded":"2024-09-14T22:49:29.944926+0000","last_fullsized":"2024-09-14T22:49:29.944926+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:59.551999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069214500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944861+0000","last_change":"2024-09-14T22:49:29.945105+0000","last_active":"2024-09-14T22:49:29.944861+0000","last_peered":"2024-09-14T22:49:29.944861+0000","last_clean":"2024-09-14T22:49:29.944861+0000","last_became_active":"2024-09-14T22:49:27.949153+0000","last_became_peered":"2024-09-14T22:49:27.949153+0000","last_unstale":"2024-09-14T22:49:29.944861+0000","last_undegraded":"2024-09-14T22:49:29.944861+0000","last_fullsized":"2024-09-14T22:49:29.944861+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:58:19.694736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000710178,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944562+0000","last_change":"2024-09-14T22:49:29.944792+0000","last_active":"2024-09-14T22:49:29.944562+0000","last_peered":"2024-09-14T22:49:29.944562+0000","last_clean":"2024-09-14T22:49:29.944562+0000","last_became_active":"2024-09-14T22:49:27.947823+0000","last_became_peered":"2024-09-14T22:49:27.947823+0000","last_unstale":"2024-09-14T22:49:29.944562+0000","last_undegraded":"2024-09-14T22:49:29.944562+0000","last_fullsized":"2024-09-14T22:49:29.944562+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:16:27.031276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043000900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.953578+0000","last_change":"2024-09-14T22:49:29.949279+0000","last_active":"2024-09-14T22:49:29.953578+0000","last_peered":"2024-09-14T22:49:29.953578+0000","last_clean":"2024-09-14T22:49:29.953578+0000","last_became_active":"2024-09-14T22:49:27.948120+0000","last_became_peered":"2024-09-14T22:49:27.948120+0000","last_unstale":"2024-09-14T22:49:29.953578+0000","last_undegraded":"2024-09-14T22:49:29.953578+0000","last_fullsized":"2024-09-14T22:49:29.953578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:53:56.777188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095602899999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.973218+0000","last_change":"2024-09-14T22:49:29.973218+0000","last_active":"2024-09-14T22:49:29.973218+0000","last_peered":"2024-09-14T22:49:29.973218+0000","last_clean":"2024-09-14T22:49:29.973218+0000","last_became_active":"2024-09-14T22:49:27.948225+0000","last_became_peered":"2024-09-14T22:49:27.948225+0000","last_unstale":"2024-09-14T22:49:29.973218+0000","last_undegraded":"2024-09-14T22:49:29.973218+0000","last_fullsized":"2024-09-14T22:49:29.973218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:34.949309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044858000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.973217+0000","last_change":"2024-09-14T22:49:29.973217+0000","last_active":"2024-09-14T22:49:29.973217+0000","last_peered":"2024-09-14T22:49:29.973217+0000","last_clean":"2024-09-14T22:49:29.973217+0000","last_became_active":"2024-09-14T22:49:27.947929+0000","last_became_peered":"2024-09-14T22:49:27.947929+0000","last_unstale":"2024-09-14T22:49:29.973217+0000","last_undegraded":"2024-09-14T22:49:29.973217+0000","last_fullsized":"2024-09-14T22:49:29.973217+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:21:03.176471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944599+0000","last_change":"2024-09-14T22:49:29.944800+0000","last_active":"2024-09-14T22:49:29.944599+0000","last_peered":"2024-09-14T22:49:29.944599+0000","last_clean":"2024-09-14T22:49:29.944599+0000","last_became_active":"2024-09-14T22:49:27.947559+0000","last_became_peered":"2024-09-14T22:49:27.947559+0000","last_unstale":"2024-09-14T22:49:29.944599+0000","last_undegraded":"2024-09-14T22:49:29.944599+0000","last_fullsized":"2024-09-14T22:49:29.944599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:33:43.576216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044691000000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T22:49:29.944243+0000","last_change":"2024-09-14T22:49:22.921000+0000","last_active":"2024-09-14T22:49:29.944243+0000","last_peered":"2024-09-14T22:49:29.944243+0000","last_clean":"2024-09-14T22:49:29.944243+0000","last_became_active":"2024-09-14T22:49:22.919993+0000","last_became_peered":"2024-09-14T22:49:22.919993+0000","last_unstale":"2024-09-14T22:49:29.944243+0000","last_undegraded":"2024-09-14T22:49:29.944243+0000","last_fullsized":"2024-09-14T22:49:29.944243+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_clean_scrub_stamp":"2024-09-14T22:49:21.912115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:51.033680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":40739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":631126,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":200704,"data_stored":631126,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T22:49:34.647 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-14T22:49:34.647 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-14T22:49:34.647 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-14T22:49:34.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-14T22:49:34.956 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T22:49:34.957 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T22:49:34.969 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-14T22:49:34.970 INFO:teuthology.run_tasks:Running task exec... 2024-09-14T22:49:34.981 INFO:teuthology.task.exec:Executing custom commands... 2024-09-14T22:49:34.981 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T22:49:34.981 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:35.331 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror] 2024-09-14T22:49:35.331 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQC0EuZm+ZNTKBAAhwAyswlu//LEJJEtTiexuA== 2024-09-14T22:49:35.331 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-14T22:49:35.331 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror 2024-09-14T22:49:35.344 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.0] 2024-09-14T22:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQC0EuZmFa6MKxAAtCEijhFxlEJiMV80z6ok6A== 2024-09-14T22:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror" 2024-09-14T22:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:35.690 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror.0 2024-09-14T22:49:35.703 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:36.054 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.1] 2024-09-14T22:49:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQC0EuZmOxAbMRAAePvxpjw6/mKJL7gFNKP7wQ== 2024-09-14T22:49:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror" 2024-09-14T22:49:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:36.054 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror.1 2024-09-14T22:49:36.068 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:36.416 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.2] 2024-09-14T22:49:36.416 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQC0EuZms5C9NhAAISuWnItpVHeymfNKSv+OFQ== 2024-09-14T22:49:36.416 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror" 2024-09-14T22:49:36.416 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:36.416 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror.2 2024-09-14T22:49:36.429 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:36.775 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.3] 2024-09-14T22:49:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQC1EuZmhSuuABAA7OAbIAua9xYCldI93hbumg== 2024-09-14T22:49:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror" 2024-09-14T22:49:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:36.776 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror.3 2024-09-14T22:49:36.788 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:37.141 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror] 2024-09-14T22:49:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZmWei+HxAAvwVOUS2eJ45Yly5d77tAFQ== 2024-09-14T22:49:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-14T22:49:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:37.142 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror 2024-09-14T22:49:37.156 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:37.506 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.0] 2024-09-14T22:49:37.507 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZmspRcJRAABkTuYS1j6xwviWPjALAnZg== 2024-09-14T22:49:37.507 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror" 2024-09-14T22:49:37.507 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:37.507 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror.0 2024-09-14T22:49:37.520 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:37.866 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.1] 2024-09-14T22:49:37.866 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZmEXMBKxAAIepEg/ldJnxCuXOtjCwA/g== 2024-09-14T22:49:37.867 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror" 2024-09-14T22:49:37.867 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:37.867 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror.1 2024-09-14T22:49:37.880 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:38.230 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.2] 2024-09-14T22:49:38.230 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZm292LMBAAJzRJDjw3ZWhG4QZTpAxM2Q== 2024-09-14T22:49:38.230 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror" 2024-09-14T22:49:38.230 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:38.230 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror.2 2024-09-14T22:49:38.244 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T22:49:38.595 INFO:teuthology.orchestra.run.smithi089.stdout:[client.mirror.3] 2024-09-14T22:49:38.595 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDZEuZm+9obNhAAzMU064b03XQ0jvmG5XMPlA== 2024-09-14T22:49:38.595 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "profile rbd-mirror" 2024-09-14T22:49:38.595 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "profile rbd" 2024-09-14T22:49:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.mirror.3 2024-09-14T22:49:38.608 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T22:49:38.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T22:49:38.620 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T22:49:38.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T22:49:38.623 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T22:49:38.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T22:49:38.632 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T22:49:38.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T22:49:38.634 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T22:49:38.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-14T22:49:38.646 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-14T22:49:38.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-14T22:49:38.648 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T22:49:38.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T22:49:38.658 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T22:49:38.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T22:49:38.660 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T22:49:38.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T22:49:38.670 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T22:49:38.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T22:49:38.673 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T22:49:38.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T22:49:38.682 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T22:49:38.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T22:49:38.684 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T22:49:38.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T22:49:38.697 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T22:49:38.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T22:49:38.699 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T22:49:38.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-14T22:49:38.709 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-14T22:49:38.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-14T22:49:38.713 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-14T22:49:38.724 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726354178 2024-09-14T22:49:38.724 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-14T22:49:38.724 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-14T22:49:38.739 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726354178 2024-09-14T22:49:38.739 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-14T22:49:38.739 INFO:teuthology.run_tasks:Running task workunit... 2024-09-14T22:49:38.753 INFO:tasks.workunit:Pulling workunits from ref ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-14T22:49:38.754 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-14T22:49:38.754 DEBUG:teuthology.orchestra.run.smithi089:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T22:49:38.755 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-14T22:49:38.755 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-14T22:49:38.755 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-14T22:49:38.755 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-09-14T22:49:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T22:49:38.775 INFO:teuthology.orchestra.run.smithi089.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-14T22:49:38.776 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T22:49:38.832 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T22:49:38.832 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-14T22:49:38.886 INFO:tasks.workunit:timeout=6h 2024-09-14T22:49:38.886 INFO:tasks.workunit:cleanup=True 2024-09-14T22:49:38.887 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-14T22:49:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-14T22:50:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Note: switching to 'ecab7be58963a5bd03910cd73eef66405d7fdd50'. 2024-09-14T22:50:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:50:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-14T22:50:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-14T22:50:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:state without impacting any branches by switching back to a branch. 2024-09-14T22:50:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:50:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-14T22:50:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-14T22:50:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:50:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: git switch -c 2024-09-14T22:50:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:50:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Or undo this operation with: 2024-09-14T22:50:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:50:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: git switch - 2024-09-14T22:50:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:50:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-14T22:50:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:50:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:HEAD is now at ecab7be5896 Merge pull request #59767 from zdover23/wip-doc-2024-09-13-backport-59438-to-squid 2024-09-14T22:50:34.319 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-14T22:50:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-14T22:50:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-14T22:50:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-14T22:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-14T22:50:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-14T22:50:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-14T22:50:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-14T22:50:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-14T22:50:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-14T22:50:35.513 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-14T22:50:35.514 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-14T22:50:35.530 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-14T22:50:35.531 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-14T22:50:35.531 DEBUG:teuthology.orchestra.run.smithi089:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ecab7be58963a5bd03910cd73eef66405d7fdd50 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-14T22:50:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ IMAGE_COUNT=50 2024-09-14T22:50:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ export LOCKDEP=0 2024-09-14T22:50:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ LOCKDEP=0 2024-09-14T22:50:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-14T22:50:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-14T22:50:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ type xmlstarlet 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ XMLSTARLET=xmlstarlet 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CLUSTER1=cluster1 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CLUSTER2=cluster2 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ POOL=mirror 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ PARENT_POOL=mirror_parent 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ NS1=ns1 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ NS2=ns2 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ TEMPDIR= 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ID=mirror 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ RBD_IMAGE_FEATURES=layering 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-14T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-14T22:50:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ MIRROR_POOL_MODE=image 2024-09-14T22:50:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' snapshot = snapshot ']' 2024-09-14T22:50:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ MIRROR_POOL_MODE=image 2024-09-14T22:50:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-14T22:50:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS='--id mirror' 2024-09-14T22:50:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-14T22:50:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-14T22:50:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-14T22:50:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-14T22:50:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_BIN=. 2024-09-14T22:50:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_SRC=. 2024-09-14T22:50:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-14T22:50:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 -gt 0 ']' 2024-09-14T22:50:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ setup 2024-09-14T22:50:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local c 2024-09-14T22:50:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-14T22:50:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ setup_tempdir 2024-09-14T22:50:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mktemp -d 2024-09-14T22:50:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ TEMPDIR=/tmp/tmp.OxLGv2e3xU 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -z 1 ']' 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ setup_pools cluster1 cluster2 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local remote_cluster=cluster2 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mon_map_file 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mon_addr 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local admin_key_file 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local uuid 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-14T22:50:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:pool 'mirror' already exists 2024-09-14T22:50:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-14T22:50:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:pool 'mirror_parent' already exists 2024-09-14T22:50:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-14T22:50:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-14T22:50:41.757 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7193428507829251) > weight (0.25) 2024-09-14T22:50:41.757 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-14T22:50:41.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-14T22:50:41.758 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-14T22:50:41.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T22:50:41.757+0000 7f3303e52640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 36865) UID: 1000 2024-09-14T22:50:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-14T22:50:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-14T22:50:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-14T22:50:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-14T22:50:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-14T22:50:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-14T22:50:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -z ']' 2024-09-14T22:50:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -z ']' 2024-09-14T22:50:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ peer_add cluster1 mirror cluster2 2024-09-14T22:50:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local client_cluster=cluster2 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local remote_cluster=cluster2 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local uuid_var_name 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local error_code 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local peer_uuid 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:50:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-14T22:50:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ peer_uuid=93ad67f3-b099-4a61-801c-ba3b3474e2be 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 -eq 0 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror_parent 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local client_cluster=cluster2 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local remote_cluster=cluster2 2024-09-14T22:50:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local uuid_var_name 2024-09-14T22:50:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local error_code 2024-09-14T22:50:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local peer_uuid 2024-09-14T22:50:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T22:50:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:50:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ peer_uuid=9240cec0-404a-4cce-9d7c-0a94e1098ac5 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 -eq 0 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ setup_pools cluster2 cluster1 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local remote_cluster=cluster1 2024-09-14T22:50:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mon_map_file 2024-09-14T22:50:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mon_addr 2024-09-14T22:50:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local admin_key_file 2024-09-14T22:50:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local uuid 2024-09-14T22:50:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-14T22:50:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:pool 'mirror' already exists 2024-09-14T22:50:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-14T22:50:45.758 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-14T22:50:45.759 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:50:45.759 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-14T22:50:45.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T22:50:45.759 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T22:50:45.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T22:50:45.762 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-09-14T22:50:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:pool 'mirror_parent' already exists 2024-09-14T22:50:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-14T22:50:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-14T22:50:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T22:50:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-14T22:50:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-14T22:50:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-14T22:50:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-14T22:50:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-14T22:50:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-14T22:50:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -z ']' 2024-09-14T22:50:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -z ']' 2024-09-14T22:50:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ peer_add cluster2 mirror cluster1 2024-09-14T22:50:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:50:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:50:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local client_cluster=cluster1 2024-09-14T22:50:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local remote_cluster=cluster1 2024-09-14T22:50:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local uuid_var_name 2024-09-14T22:50:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local error_code 2024-09-14T22:50:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local peer_uuid 2024-09-14T22:50:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T22:50:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:50:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-14T22:50:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ peer_uuid=bc7e50cb-c4cc-46bd-97c5-88bbdad1ce7b 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 -eq 0 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror_parent 2024-09-14T22:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local client_cluster=cluster1 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local remote_cluster=cluster1 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local uuid_var_name 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local error_code 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local peer_uuid 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:50:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-14T22:50:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ peer_uuid=02c81f47-4976-4428-9c58-0c4ded380d41 2024-09-14T22:50:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T22:50:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:50:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T22:50:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 -eq 0 2024-09-14T22:50:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:50:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:50:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirrors cluster1 2024-09-14T22:50:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:50:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 0 3 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirror cluster1:0 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1:0 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local val=cluster1:0 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster_var_name=cluster 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance_var_name=instance 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cluster=cluster1 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=0 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval cluster=cluster1 2024-09-14T22:50:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval instance=0 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirror cluster1:1 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1:1 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local val=cluster1:1 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster_var_name=cluster 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance_var_name=instance 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cluster=cluster1 2024-09-14T22:50:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=1 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval cluster=cluster1 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval instance=1 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirror cluster1:2 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1:2 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local val=cluster1:2 2024-09-14T22:50:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster_var_name=cluster 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance_var_name=instance 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cluster=cluster1 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=2 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval cluster=cluster1 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval instance=2 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirror cluster1:3 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1:3 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:50:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local val=cluster1:3 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster_var_name=cluster 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance_var_name=instance 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cluster=cluster1 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=3 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval cluster=cluster1 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval instance=3 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirrors cluster2 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:50:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 0 3 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirror cluster2:0 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2:0 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local val=cluster2:0 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster_var_name=cluster 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance_var_name=instance 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cluster=cluster2 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=0 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval cluster=cluster2 2024-09-14T22:50:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster2 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval instance=0 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirror cluster2:1 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2:1 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local val=cluster2:1 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster_var_name=cluster 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance_var_name=instance 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cluster=cluster2 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=1 2024-09-14T22:50:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval cluster=cluster2 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster2 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval instance=1 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirror cluster2:2 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2:2 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local val=cluster2:2 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster_var_name=cluster 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance_var_name=instance 2024-09-14T22:50:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cluster=cluster2 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=2 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval cluster=cluster2 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster2 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval instance=2 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirror cluster2:3 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2:3 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-14T22:50:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local val=cluster2:3 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster_var_name=cluster 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance_var_name=instance 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cluster=cluster2 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=3 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval cluster=cluster2 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster2 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval instance=3 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ tee -a /tmp/tmp.OxLGv2e3xU/rbd-mirror.test.log 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ date '+%F %T' 2024-09-14T22:50:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo 2024-09-14 22:50:52 TEST: add image and test replay after client crashes 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:2024-09-14 22:50:52 TEST: add image and test replay after client crashes 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=test 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror test 512M 2024-09-14T22:50:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T22:50:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T22:50:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-14T22:50:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T22:50:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-14T22:50:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:50:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10b09a7d639b 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10b09a7d639b 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:50:52 2024 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:50:52 2024 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:50:52 2024 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:50:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:50:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T22:50:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:50:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:50:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:50:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:50:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:50:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:50:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:50:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:50:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:50:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:50:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:50:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:50:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:50:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:50:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:50:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:50:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:50:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:50:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:50:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:50:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:50:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:50:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:50:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:50:54.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:50:54.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:50:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:50:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:50:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:50:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:50:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:50:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:50:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:50:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:50:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:50:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:50:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:50:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:50:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:50:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:50:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:50:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:50:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:50:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:50:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:50:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:50:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:50:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 1 2024-09-14T22:50:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:50:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:50:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:50:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:50:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:50:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:50:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:50:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:50:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:50:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:50:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:50:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:50:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:50:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:50:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:50:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:50:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:50:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:50:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:50:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:50:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:50:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:50:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:50:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:50:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:50:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:50:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:50:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:50:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:50:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:50:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 2 2024-09-14T22:50:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:50:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:50:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:50:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:50:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:50:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:50:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:50:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:50:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:50:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:50:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:50:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:50:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:50:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:50:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:50:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:50:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:50:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:50:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:50:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:50:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:50:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:50:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:50:58.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:50:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:50:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:50:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:50:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:50:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:50:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:50:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:50:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:50:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:50:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:50:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:50:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:50:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:50:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:50:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:50:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 4 2024-09-14T22:51:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:51:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:51:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:51:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:51:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:51:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:51:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:51:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:51:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:51:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:51:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:51:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:51:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:51:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:51:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:51:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:51:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:51:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:51:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:51:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:51:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:51:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:51:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:51:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:51:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:51:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:51:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:51:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:51:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:51:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:51:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:51:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:51:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:51:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:51:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:51:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:51:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:51:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:51:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:51:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:51:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:51:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:51:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:51:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T22:51:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:51:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:51:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:51:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:51:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:51:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:51:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:51:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:51:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:51:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:51:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:51:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:51:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:51:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:51:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:51:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:51:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:51:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:51:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:51:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:51:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:51:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:51:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:51:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:51:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:51:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:51:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:51:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:51:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:51:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:51:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:51:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:51:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:51:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:51:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:51:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:51:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:51:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:51:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:51:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:51:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:51:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:51:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:51:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:51:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T22:51:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:51:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:51:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:51:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:51:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:51:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:51:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:51:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:51:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:51:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:51:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:51:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:51:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:51:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:51:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:51:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name= 2024-09-14T22:51:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 10 2024-09-14T22:51:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:51:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:51:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:51:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:51:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=5 2024-09-14T22:51:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:51:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:51:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 79456256 bytes 2024-09-14T22:51:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 123301888 bytes 2024-09-14T22:51:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 176356352 bytes 2024-09-14T22:51:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 191812608 bytes 2024-09-14T22:51:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 202716160 bytes 2024-09-14T22:51:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 205701120 bytes 2024-09-14T22:51:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 289334272 bytes 2024-09-14T22:51:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 303351808 bytes 2024-09-14T22:51:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 314461184 bytes 2024-09-14T22:51:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 395557888 bytes 2024-09-14T22:51:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 420965376 bytes 2024-09-14T22:51:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 468869120 bytes 2024-09-14T22:51:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 506591232 bytes 2024-09-14T22:51:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 517574656 bytes 2024-09-14T22:51:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 592936960 bytes 2024-09-14T22:51:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 622510080 bytes 2024-09-14T22:51:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 668920832 bytes 2024-09-14T22:51:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 690010112 bytes 2024-09-14T22:51:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 726131712 bytes 2024-09-14T22:51:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 754832384 bytes 2024-09-14T22:51:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 790957056 bytes 2024-09-14T22:51:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 824042496 bytes 2024-09-14T22:51:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 839236608 bytes 2024-09-14T22:51:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 909710336 bytes 2024-09-14T22:51:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 971961344 bytes 2024-09-14T22:51:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1017870336 bytes 2024-09-14T22:51:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1043846144 bytes 2024-09-14T22:51:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1069547520 bytes 2024-09-14T22:51:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1112313856 bytes 2024-09-14T22:51:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1171388416 bytes 2024-09-14T22:51:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1194255360 bytes 2024-09-14T22:51:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1272176640 bytes 2024-09-14T22:51:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1279839232 bytes 2024-09-14T22:51:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1331375104 bytes 2024-09-14T22:51:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1390602240 bytes 2024-09-14T22:51:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1448714240 bytes 2024-09-14T22:51:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1455918080 bytes 2024-09-14T22:51:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1456536576 bytes 2024-09-14T22:51:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1496324096 bytes 2024-09-14T22:51:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1587490816 bytes 2024-09-14T22:51:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1632754688 bytes 2024-09-14T22:51:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1633367040 bytes 2024-09-14T22:51:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1707505664 bytes 2024-09-14T22:51:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1782867968 bytes 2024-09-14T22:51:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1818917888 bytes 2024-09-14T22:51:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1872730112 bytes 2024-09-14T22:51:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1873342464 bytes 2024-09-14T22:51:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1894981632 bytes 2024-09-14T22:51:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1935408128 bytes 2024-09-14T22:51:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1994772480 bytes 2024-09-14T22:51:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2013409280 bytes 2024-09-14T22:51:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2052393984 bytes 2024-09-14T22:51:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2113660928 bytes 2024-09-14T22:51:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2126505984 bytes 2024-09-14T22:51:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2138884096 bytes 2024-09-14T22:51:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2182923264 bytes 2024-09-14T22:51:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2183561216 bytes 2024-09-14T22:51:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2199097344 bytes 2024-09-14T22:51:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2232109056 bytes 2024-09-14T22:51:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:51:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:51:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:51:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:51:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:51:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:51:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:51:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: down+unknown 2024-09-14T22:51:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: status not found 2024-09-14T22:51:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14T22:51:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:51:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:51:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: down+unknown 2024-09-14T22:51:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: status not found 2024-09-14T22:51:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: ' 2024-09-14T22:51:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:51:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 1 2024-09-14T22:51:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:51:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T22:51:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:51:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:51:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:51:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:51:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: down+unknown 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: status not found 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: down+unknown 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: status not found 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: ' 2024-09-14T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:51:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 1 2024-09-14T22:51:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:51:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T22:51:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:51:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:51:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:51:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:51:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:51:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:51:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: down+unknown 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: status not found 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: down+unknown 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: status not found 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: ' 2024-09-14T22:51:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:51:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 1 2024-09-14T22:51:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:51:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T22:51:35.757 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5285550208098267) > weight (0.25) 2024-09-14T22:51:35.757 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8194016265486262) > weight (0.25) 2024-09-14T22:51:35.757 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-14T22:51:35.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-14T22:51:35.758 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T22:51:35.758 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-14T22:51:35.758 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:51:35.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T22:51:35.757+0000 7f2f84e38640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 36816) UID: 1000 2024-09-14T22:51:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:51:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:51:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:51:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:51:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:51:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:51:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:51:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:51:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:51:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:51:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354253,"remote_snapshot_timestamp":1726354253,"replay_state":"idle"} 2024-09-14T22:51:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.0 on smithi089 2024-09-14T22:51:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:51:38 2024-09-14T22:51:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:51:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:51:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+stopped 2024-09-14T22:51:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: local image is primary 2024-09-14T22:51:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:51:38' 2024-09-14T22:51:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:51:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:51:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:51:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354253,"remote_snapshot_timestamp":1726354253,"replay_state":"idle"} 2024-09-14T22:51:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:51:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:51:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:51:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:51:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:51:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.0 on smithi089 2024-09-14T22:51:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:51:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:51:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:51:38' 2024-09-14T22:51:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:51:38' 2024-09-14T22:51:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:51:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:51:38' 2024-09-14T22:51:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:51:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:51:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:51:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='OKOKOK142084259mirror.0smithi08919.1.1-324-gecab7be5falseOK42124262mirror.1smithi08919.1.1-324-gecab7be5trueOK42204268mirror.3smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354253,"remote_snapshot_timestamp":1726354253,"replay_state":"idle"}42084259mirror.0smithi0892024-09-14 22:51:38cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38up+stoppedlocal image is primary2024-09-14 22:51:38' 2024-09-14T22:51:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:51:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:51:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:51:38' 2024-09-14T22:51:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:51:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:51:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:51:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:51:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap1 2024-09-14T22:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap1 2024-09-14T22:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap1 2024-09-14T22:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:51:41.860 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-14T22:51:41.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-14T22:51:41.861 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-14T22:51:41.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-14T22:51:41.864 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:51:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:51:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:51:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:51:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:51:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:51:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:51:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:51:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:51:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:51:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:51:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:51:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:51:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:51:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:51:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:51:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:51:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:51:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:51:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:51:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:51:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:51:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:51:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:51:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:51:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:51:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:51:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:51:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:51:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:51:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:51:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:51:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:51:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:51:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:51:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:51:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:51:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:51:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:51:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:51:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:51:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:51:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:51:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:51:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:51:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:51:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:51:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:51:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:51:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 5 2024-09-14T22:51:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:51:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:51:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:51:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:51:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:51:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:51:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=5 2024-09-14T22:51:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:51:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:51:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:51:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:51:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:51:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:51:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:51:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:51:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=3 2024-09-14T22:51:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 5 = 3 2024-09-14T22:51:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:51:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:51:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:51:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:51:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:51:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:51:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:51:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:51:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:51:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:51:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:51:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:51:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 5 = '' 2024-09-14T22:51:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:51:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:51:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:51:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:51:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:51:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:51:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:51:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:51:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:51:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:51:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:51:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:51:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 5 = '' 2024-09-14T22:51:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:51:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:51:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:51:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:51:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:51:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:51:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:51:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=5 2024-09-14T22:51:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 5 = 5 2024-09-14T22:51:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:51:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-14T22:51:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:51:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:51:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:51:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap1 2024-09-14T22:51:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:51:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:51:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:52:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:52:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:52:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:52:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:52:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:52:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:52:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:52:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:52:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap1 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap1 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:52:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:52:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:52:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap1-clean 2024-09-14T22:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-14T22:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap1-clean 2024-09-14T22:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:52:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:52:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:52:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:52:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=5 2024-09-14T22:52:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:52:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:52:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 4993024 bytes 2024-09-14T22:52:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 31430656 bytes 2024-09-14T22:52:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 137102336 bytes 2024-09-14T22:52:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 152018944 bytes 2024-09-14T22:52:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 248562688 bytes 2024-09-14T22:52:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 309234688 bytes 2024-09-14T22:52:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 373850112 bytes 2024-09-14T22:52:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 442814464 bytes 2024-09-14T22:52:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 482832384 bytes 2024-09-14T22:52:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 541060096 bytes 2024-09-14T22:52:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 633413632 bytes 2024-09-14T22:52:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 711775232 bytes 2024-09-14T22:52:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 785108992 bytes 2024-09-14T22:52:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 836322304 bytes 2024-09-14T22:52:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 878455808 bytes 2024-09-14T22:52:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 879086592 bytes 2024-09-14T22:52:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 941913088 bytes 2024-09-14T22:52:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 945460224 bytes 2024-09-14T22:52:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 990735360 bytes 2024-09-14T22:52:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 1023461376 bytes 2024-09-14T22:52:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 1091908608 bytes 2024-09-14T22:52:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1092515840 bytes 2024-09-14T22:52:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1130716160 bytes 2024-09-14T22:52:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1159482368 bytes 2024-09-14T22:52:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1208157184 bytes 2024-09-14T22:52:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1221987328 bytes 2024-09-14T22:52:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1240283136 bytes 2024-09-14T22:52:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1256469504 bytes 2024-09-14T22:52:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1313928192 bytes 2024-09-14T22:52:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1342876672 bytes 2024-09-14T22:52:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1383982080 bytes 2024-09-14T22:52:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1435102208 bytes 2024-09-14T22:52:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1522941952 bytes 2024-09-14T22:52:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1552568320 bytes 2024-09-14T22:52:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1593352192 bytes 2024-09-14T22:52:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1642294272 bytes 2024-09-14T22:52:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1672779776 bytes 2024-09-14T22:52:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1700524032 bytes 2024-09-14T22:52:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1732860928 bytes 2024-09-14T22:52:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1786189824 bytes 2024-09-14T22:52:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1829511168 bytes 2024-09-14T22:52:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1862886400 bytes 2024-09-14T22:52:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1893169152 bytes 2024-09-14T22:52:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1919382528 bytes 2024-09-14T22:52:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1997545472 bytes 2024-09-14T22:52:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2022663168 bytes 2024-09-14T22:52:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2025742336 bytes 2024-09-14T22:52:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2110015488 bytes 2024-09-14T22:52:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2181514240 bytes 2024-09-14T22:52:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2230500352 bytes 2024-09-14T22:52:06.763 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5696659000016787) > weight (0.25) 2024-09-14T22:52:06.764 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-14T22:52:06.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-14T22:52:06.764 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T22:52:06.764 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-14T22:52:06.764 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:52:06.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T22:52:06.764+0000 7fda8b3c8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 37570) UID: 1000 2024-09-14T22:52:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2334441472 bytes 2024-09-14T22:52:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2348759040 bytes 2024-09-14T22:52:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:52:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:52:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:52:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:52:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:52:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:52:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:52:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354253,"remote_snapshot_timestamp":1726354253,"replay_state":"idle"} 2024-09-14T22:52:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.0 on smithi089 2024-09-14T22:52:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:51:38 2024-09-14T22:52:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:52:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:52:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+stopped 2024-09-14T22:52:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: local image is primary 2024-09-14T22:52:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:51:38' 2024-09-14T22:52:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:52:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:52:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:52:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354253,"remote_snapshot_timestamp":1726354253,"replay_state":"idle"} 2024-09-14T22:52:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:52:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:52:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:52:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:52:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:52:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.0 on smithi089 2024-09-14T22:52:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:52:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:52:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:51:38' 2024-09-14T22:52:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:51:38' 2024-09-14T22:52:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:52:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:51:38' 2024-09-14T22:52:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:52:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:52:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:52:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='OKOKOK142084259mirror.0smithi08919.1.1-324-gecab7be5falseOK42124262mirror.1smithi08919.1.1-324-gecab7be5trueOK42204268mirror.3smithi08919.1.1-324-gecab7be5falseOK43224325mirror.2smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354253,"remote_snapshot_timestamp":1726354253,"replay_state":"idle"}42084259mirror.0smithi0892024-09-14 22:51:38cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38up+stoppedlocal image is primary2024-09-14 22:51:38' 2024-09-14T22:52:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:52:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:52:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:51:38' 2024-09-14T22:52:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:52:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:52:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:52:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:52:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:52:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap2 2024-09-14T22:52:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap2 2024-09-14T22:52:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap2 2024-09-14T22:52:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:52:12.866 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-14T22:52:12.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T22:52:12.866 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T22:52:12.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T22:52:12.870 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-14T22:52:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:52:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:52:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:52:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:52:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:52:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:52:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:52:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:52:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:52:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:52:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:52:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:52:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:52:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:52:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:52:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:52:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:52:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:52:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:52:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:52:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:52:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:52:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:52:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:52:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:52:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:52:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:52:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:52:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:52:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:52:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:52:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:52:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:52:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:52:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:52:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:52:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:52:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:52:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 8 2024-09-14T22:52:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:52:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:52:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:52:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:52:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:52:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=8 2024-09-14T22:52:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:52:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:52:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:52:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:52:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:52:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:52:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:52:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:52:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:52:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:52:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:52:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:52:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:52:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 8 = '' 2024-09-14T22:52:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:52:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:52:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:52:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:52:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:52:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:52:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:52:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:52:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:52:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:52:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:52:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:52:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 8 = '' 2024-09-14T22:52:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:52:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:52:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:52:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:52:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:52:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:52:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:52:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:52:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:52:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:52:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:52:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:52:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 8 = '' 2024-09-14T22:52:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:52:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:52:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:52:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:52:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:52:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:52:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=8 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 8 = 8 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap2 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:52:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:52:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-14T22:52:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:52:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:52:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:52:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 4 2024-09-14T22:52:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:52:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:52:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:52:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:52:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snap1-clean ']' 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap1-clean 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:52:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:52:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:52:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap2 2024-09-14T22:52:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap2 2024-09-14T22:52:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:52:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:52:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:52:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:52:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:52:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:52:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap2-clean 2024-09-14T22:52:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-14T22:52:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap2-clean 2024-09-14T22:52:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:52:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:52:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:52:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:52:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=10 2024-09-14T22:52:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:52:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:52:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 27133952 bytes 2024-09-14T22:52:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 88179712 bytes 2024-09-14T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 129080320 bytes 2024-09-14T22:52:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 139846656 bytes 2024-09-14T22:52:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 152283136 bytes 2024-09-14T22:52:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 178681856 bytes 2024-09-14T22:52:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 254649344 bytes 2024-09-14T22:52:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 296762368 bytes 2024-09-14T22:52:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 383377408 bytes 2024-09-14T22:52:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 440425472 bytes 2024-09-14T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 514161664 bytes 2024-09-14T22:52:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 527022080 bytes 2024-09-14T22:52:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 580511744 bytes 2024-09-14T22:52:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 601270272 bytes 2024-09-14T22:52:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 676640768 bytes 2024-09-14T22:52:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 712770560 bytes 2024-09-14T22:52:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 771261440 bytes 2024-09-14T22:52:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 816441344 bytes 2024-09-14T22:52:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 876969984 bytes 2024-09-14T22:52:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 927938560 bytes 2024-09-14T22:52:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 948426752 bytes 2024-09-14T22:52:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1047316480 bytes 2024-09-14T22:52:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1093049344 bytes 2024-09-14T22:52:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1130583040 bytes 2024-09-14T22:52:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1169703936 bytes 2024-09-14T22:52:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1191690240 bytes 2024-09-14T22:52:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1279811584 bytes 2024-09-14T22:52:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1299240960 bytes 2024-09-14T22:52:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1371710464 bytes 2024-09-14T22:52:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1413080064 bytes 2024-09-14T22:52:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1493174272 bytes 2024-09-14T22:52:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1570068480 bytes 2024-09-14T22:52:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1594290176 bytes 2024-09-14T22:52:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1607393280 bytes 2024-09-14T22:52:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1713268736 bytes 2024-09-14T22:52:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1725172736 bytes 2024-09-14T22:52:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1725784064 bytes 2024-09-14T22:52:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1780838400 bytes 2024-09-14T22:52:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1843162112 bytes 2024-09-14T22:52:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1873807360 bytes 2024-09-14T22:52:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1927806976 bytes 2024-09-14T22:52:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1993418752 bytes 2024-09-14T22:52:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 2030616576 bytes 2024-09-14T22:52:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 2113930240 bytes 2024-09-14T22:52:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2166975488 bytes 2024-09-14T22:52:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2167589888 bytes 2024-09-14T22:52:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2198193152 bytes 2024-09-14T22:52:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2203151360 bytes 2024-09-14T22:52:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2252742656 bytes 2024-09-14T22:52:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2289913856 bytes 2024-09-14T22:52:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2367213568 bytes 2024-09-14T22:52:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2449068032 bytes 2024-09-14T22:52:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2479748096 bytes 2024-09-14T22:52:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2527048704 bytes 2024-09-14T22:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2572381184 bytes 2024-09-14T22:52:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2586482688 bytes 2024-09-14T22:52:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2655531008 bytes 2024-09-14T22:52:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2730606592 bytes 2024-09-14T22:52:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2821321728 bytes 2024-09-14T22:52:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2849744896 bytes 2024-09-14T22:52:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2910423040 bytes 2024-09-14T22:52:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2983614464 bytes 2024-09-14T22:52:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 3007143936 bytes 2024-09-14T22:52:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 3049543680 bytes 2024-09-14T22:52:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 3053635584 bytes 2024-09-14T22:52:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 3111244800 bytes 2024-09-14T22:52:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 3135932416 bytes 2024-09-14T22:52:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 3170859008 bytes 2024-09-14T22:52:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3188074496 bytes 2024-09-14T22:52:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3259036672 bytes 2024-09-14T22:52:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3355985920 bytes 2024-09-14T22:52:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3356743680 bytes 2024-09-14T22:52:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3387670528 bytes 2024-09-14T22:52:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3475733504 bytes 2024-09-14T22:52:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3537042432 bytes 2024-09-14T22:52:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3594312704 bytes 2024-09-14T22:52:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3623111680 bytes 2024-09-14T22:52:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3668641792 bytes 2024-09-14T22:52:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3680211968 bytes 2024-09-14T22:52:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3714210816 bytes 2024-09-14T22:52:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3753847808 bytes 2024-09-14T22:52:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3774605312 bytes 2024-09-14T22:52:55.866 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3631074225678228) > weight (0.25) 2024-09-14T22:52:55.866 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7362870866138485) > weight (0.25) 2024-09-14T22:52:55.866 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.35608961936407535) > weight (0.25) 2024-09-14T22:52:55.866 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-14T22:52:55.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-14T22:52:55.866 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-14T22:52:55.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T22:52:55.868+0000 7f6200b30640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 36832) UID: 1000 2024-09-14T22:52:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3844523008 bytes 2024-09-14T22:52:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3914350592 bytes 2024-09-14T22:52:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3965061120 bytes 2024-09-14T22:52:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 4011384832 bytes 2024-09-14T22:52:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 4053697536 bytes 2024-09-14T22:52:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 4088503296 bytes 2024-09-14T22:52:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 4111175680 bytes 2024-09-14T22:52:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 4120491008 bytes 2024-09-14T22:52:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 4142840832 bytes 2024-09-14T22:52:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 4196925440 bytes 2024-09-14T22:52:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 4236050432 bytes 2024-09-14T22:52:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4289262592 bytes 2024-09-14T22:52:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4406760448 bytes 2024-09-14T22:52:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4459043840 bytes 2024-09-14T22:52:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4459648000 bytes 2024-09-14T22:52:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4520702976 bytes 2024-09-14T22:52:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4574212096 bytes 2024-09-14T22:52:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4586103808 bytes 2024-09-14T22:52:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4648313856 bytes 2024-09-14T22:52:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4767831040 bytes 2024-09-14T22:52:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4785883136 bytes 2024-09-14T22:52:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4811499520 bytes 2024-09-14T22:52:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:52:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:52:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:52:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:52:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:52:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:52:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:52:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:52:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:52:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:52:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:52:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:52:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:52:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:52:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:52:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:52:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:52:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:52:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:52:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":256960000.0,"last_snapshot_bytes":513920000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354315,"remote_snapshot_timestamp":1726354315,"replay_state":"idle"} 2024-09-14T22:52:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.0 on smithi089 2024-09-14T22:52:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:52:38 2024-09-14T22:52:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:52:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:52:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+stopped 2024-09-14T22:52:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: local image is primary 2024-09-14T22:52:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:52:38' 2024-09-14T22:52:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:52:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:52:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:52:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":256960000.0,"last_snapshot_bytes":513920000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354315,"remote_snapshot_timestamp":1726354315,"replay_state":"idle"} 2024-09-14T22:52:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:52:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:52:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:52:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:52:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:52:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.0 on smithi089 2024-09-14T22:52:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:52:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:52:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:52:38' 2024-09-14T22:52:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:52:38' 2024-09-14T22:52:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:52:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:52:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:52:38' 2024-09-14T22:52:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:52:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:52:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:52:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='OKOKOK142084259mirror.0smithi08919.1.1-324-gecab7be5falseOK42124262mirror.1smithi08919.1.1-324-gecab7be5trueOK43224325mirror.2smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":256960000.0,"last_snapshot_bytes":513920000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354315,"remote_snapshot_timestamp":1726354315,"replay_state":"idle"}42084259mirror.0smithi0892024-09-14 22:52:38cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38up+stoppedlocal image is primary2024-09-14 22:52:38' 2024-09-14T22:52:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:52:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:52:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:52:38' 2024-09-14T22:52:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:52:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:52:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap3 2024-09-14T22:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap3 2024-09-14T22:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap3 2024-09-14T22:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:53:04.868 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-14T22:53:04.868 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:53:04.869 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-14T22:53:04.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T22:53:04.869 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T22:53:04.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T22:53:04.872 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-14T22:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:53:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:53:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:53:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:53:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:53:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:53:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:53:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:53:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:53:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:53:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:53:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:53:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:53:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:53:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:53:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:53:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:53:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:53:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:53:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:53:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:53:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:53:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:53:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:53:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:53:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:53:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:53:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:53:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:53:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:53:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:53:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:53:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:53:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 11 2024-09-14T22:53:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:53:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:53:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:53:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:53:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:53:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:53:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=11 2024-09-14T22:53:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:53:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:53:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:53:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:53:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:53:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:53:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:53:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:53:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:53:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:53:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:53:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:53:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:53:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 11 = '' 2024-09-14T22:53:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:53:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:53:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:53:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:53:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:53:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:53:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:53:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:53:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:53:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:53:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:53:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:53:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 11 = '' 2024-09-14T22:53:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:53:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:53:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:53:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:53:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:53:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:53:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:53:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:53:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:53:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:53:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:53:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:53:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 11 = '' 2024-09-14T22:53:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:53:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:53:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:53:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:53:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:53:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:53:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:53:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=11 2024-09-14T22:53:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 11 = 11 2024-09-14T22:53:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:53:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-14T22:53:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap3 2024-09-14T22:53:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:53:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:53:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:53:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-14T22:53:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:53:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:53:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:53:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 6 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:52:44 2024 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:53:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:53:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:53:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:53:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:53:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snap2-clean ']' 2024-09-14T22:53:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-14T22:53:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap2-clean 2024-09-14T22:53:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:53:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:53:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:53:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:53:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:53:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:53:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap3 2024-09-14T22:53:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap3 2024-09-14T22:53:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:53:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:53:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:53:37.870 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7352546154134545) > weight (0.25) 2024-09-14T22:53:37.870 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-14T22:53:37.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-14T22:53:37.871 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T22:53:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:53:37.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T22:53:37.870+0000 7f66e93dd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 39291) UID: 1000 2024-09-14T22:53:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:53:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:53:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:53:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap3-clean 2024-09-14T22:53:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-14T22:53:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:53:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap3-clean 2024-09-14T22:53:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:53:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:53:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:53:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:53:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:53:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=5 2024-09-14T22:53:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:53:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:53:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 37664768 bytes 2024-09-14T22:53:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 78300160 bytes 2024-09-14T22:53:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 110460928 bytes 2024-09-14T22:53:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 111073280 bytes 2024-09-14T22:53:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 172912640 bytes 2024-09-14T22:53:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 238875648 bytes 2024-09-14T22:53:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 264723456 bytes 2024-09-14T22:53:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 267217920 bytes 2024-09-14T22:53:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 336709632 bytes 2024-09-14T22:53:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 351576064 bytes 2024-09-14T22:53:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 430134272 bytes 2024-09-14T22:53:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 451264512 bytes 2024-09-14T22:53:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 493394944 bytes 2024-09-14T22:53:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 502999040 bytes 2024-09-14T22:53:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 567469056 bytes 2024-09-14T22:53:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 596608000 bytes 2024-09-14T22:53:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 694491136 bytes 2024-09-14T22:53:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 695109632 bytes 2024-09-14T22:53:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 730559488 bytes 2024-09-14T22:53:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 750827520 bytes 2024-09-14T22:53:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 763867136 bytes 2024-09-14T22:53:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 789051392 bytes 2024-09-14T22:53:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 860037120 bytes 2024-09-14T22:53:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 892195840 bytes 2024-09-14T22:53:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 976996352 bytes 2024-09-14T22:53:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 994640896 bytes 2024-09-14T22:53:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1040860160 bytes 2024-09-14T22:53:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1118489600 bytes 2024-09-14T22:53:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1145211904 bytes 2024-09-14T22:53:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1145831424 bytes 2024-09-14T22:53:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1205299200 bytes 2024-09-14T22:53:42.871 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-14T22:53:42.872 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:53:42.872 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-14T22:53:42.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T22:53:42.872 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T22:53:42.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T22:53:42.875 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-14T22:53:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1263977472 bytes 2024-09-14T22:53:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1322955776 bytes 2024-09-14T22:53:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1378716672 bytes 2024-09-14T22:53:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1385263104 bytes 2024-09-14T22:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1422691328 bytes 2024-09-14T22:53:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1442445312 bytes 2024-09-14T22:53:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1448259584 bytes 2024-09-14T22:53:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1525499904 bytes 2024-09-14T22:53:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1601972224 bytes 2024-09-14T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1602592768 bytes 2024-09-14T22:53:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1642051584 bytes 2024-09-14T22:53:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1672599552 bytes 2024-09-14T22:53:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1719855104 bytes 2024-09-14T22:53:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1781454848 bytes 2024-09-14T22:53:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1813704704 bytes 2024-09-14T22:53:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1816422400 bytes 2024-09-14T22:53:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1817028608 bytes 2024-09-14T22:53:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1855858688 bytes 2024-09-14T22:53:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1891279872 bytes 2024-09-14T22:53:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 1895224320 bytes 2024-09-14T22:53:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 1961673728 bytes 2024-09-14T22:53:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 1999311872 bytes 2024-09-14T22:53:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2040600576 bytes 2024-09-14T22:53:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2063597568 bytes 2024-09-14T22:53:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2077248512 bytes 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:53:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:53:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:53:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:53:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:53:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:53:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:53:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:53:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:53:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:53:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:53:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:53:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:53:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:53:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":17888699.73,"bytes_per_snapshot":531910656.0,"last_snapshot_bytes":536660992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354410,"remote_snapshot_timestamp":1726354410,"replay_state":"idle"} 2024-09-14T22:53:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.0 on smithi089 2024-09-14T22:53:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:53:38 2024-09-14T22:53:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:53:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:53:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+stopped 2024-09-14T22:53:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: local image is primary 2024-09-14T22:53:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:53:38' 2024-09-14T22:53:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:53:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:53:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:53:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":17888699.73,"bytes_per_snapshot":531910656.0,"last_snapshot_bytes":536660992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354410,"remote_snapshot_timestamp":1726354410,"replay_state":"idle"} 2024-09-14T22:53:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:53:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:53:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:53:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:53:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:53:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.0 on smithi089 2024-09-14T22:53:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:53:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:53:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:53:38' 2024-09-14T22:53:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:53:38' 2024-09-14T22:53:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:53:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:53:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:53:38' 2024-09-14T22:53:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:53:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:53:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:53:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='OKOKOK142084259mirror.0smithi08919.1.1-324-gecab7be5falseOK42124262mirror.1smithi08919.1.1-324-gecab7be5trueOK43224325mirror.2smithi08919.1.1-324-gecab7be5falseOK43874390mirror.3smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":17888699.73,"bytes_per_snapshot":531910656.0,"last_snapshot_bytes":536660992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354410,"remote_snapshot_timestamp":1726354410,"replay_state":"idle"}42084259mirror.0smithi0892024-09-14 22:53:38cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38up+stoppedlocal image is primary2024-09-14 22:53:38' 2024-09-14T22:53:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:53:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:53:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:53:38' 2024-09-14T22:53:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:53:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:53:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:53:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:53:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:53:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:53:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:53:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap4 2024-09-14T22:53:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap4 2024-09-14T22:53:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:53:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:53:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:53:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap4 2024-09-14T22:53:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:54:11.873 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T22:54:11.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T22:54:11.874 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-14T22:54:11.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T22:54:11.874+0000 7fafaad8f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36782) UID: 1000 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:54:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:54:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:54:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:54:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:54:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:54:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:54:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:54:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:54:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:54:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:54:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:54:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:54:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:54:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:54:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:54:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:54:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:54:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:54:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:54:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:54:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:54:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:54:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:54:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:54:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:54:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:54:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:54:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:54:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:54:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:54:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:54:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:54:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:54:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:54:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 1 2024-09-14T22:54:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:54:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:54:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:54:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:54:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:54:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:54:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:54:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:54:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:54:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:54:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:54:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:54:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:54:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:54:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:54:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:54:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:54:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:54:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:54:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:54:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:54:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:54:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:54:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 2 2024-09-14T22:54:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:54:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:54:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:54:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:54:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:54:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:54:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:54:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:54:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:54:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:54:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:54:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:54:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:54:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:54:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:54:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:54:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:54:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:54:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:54:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:54:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:54:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:54:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 4 2024-09-14T22:54:19.875 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T22:54:19.875 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:54:19.876 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T22:54:19.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T22:54:19.876 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T22:54:19.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T22:54:19.881 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-14T22:54:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:54:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:54:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:54:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:54:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:54:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:54:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:54:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:54:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:54:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:54:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:54:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:54:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:54:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:54:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:54:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:54:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:54:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:54:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:54:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:54:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:54:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:54:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:54:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:54:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:54:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:54:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:54:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T22:54:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:54:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:54:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:54:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:54:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:54:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:54:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:54:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:54:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:54:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:54:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:54:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:54:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:54:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:54:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:54:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:54:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:54:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:54:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:54:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:54:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:54:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:54:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:54:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:54:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:54:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T22:54:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:54:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:54:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:54:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:54:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:54:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:54:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:54:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:54:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:54:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:54:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:54:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:54:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:54:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:54:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:54:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:54:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:54:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:54:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:54:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:54:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:54:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:54:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T22:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T22:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T22:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T22:54:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T22:54:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T22:54:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T22:54:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T22:54:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T22:54:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T22:54:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T22:54:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T22:54:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T22:54:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T22:54:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:54:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T22:54:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:54:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:54:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:54:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:54:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:54:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:54:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:54:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:54:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:54:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:54:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:54:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:54:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:54:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:54:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:54:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:54:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:54:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:54:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:54:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:54:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:54:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:54:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:54:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:54:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:54:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:54:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:54:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:54:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:54:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:54:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:54:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:54:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:54:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:54:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 14 2024-09-14T22:54:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:54:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:54:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:54:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:54:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:54:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:54:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=14 2024-09-14T22:54:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:54:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:54:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:54:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:54:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:54:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:54:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:54:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:54:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:54:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:54:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:54:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:54:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:54:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:54:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 14 = '' 2024-09-14T22:54:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:54:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:54:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:54:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:54:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:54:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:54:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:54:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:54:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:54:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:54:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:54:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:54:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:54:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 14 = '' 2024-09-14T22:54:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:54:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:54:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:54:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:54:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:54:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:54:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:54:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:54:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:54:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:54:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:54:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:54:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:54:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 14 = '' 2024-09-14T22:54:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:54:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:54:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:54:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:54:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:54:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:54:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:54:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:54:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=14 2024-09-14T22:54:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 14 = 14 2024-09-14T22:54:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-14T22:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap4 2024-09-14T22:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:54:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-14T22:54:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:54:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:54:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:54:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 8 2024-09-14T22:54:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:54:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:54:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:52:44 2024 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:54:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:54:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snap3-clean ']' 2024-09-14T22:54:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-14T22:54:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap3-clean 2024-09-14T22:54:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:54:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:54:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:54:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:54:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:54:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:54:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap4 2024-09-14T22:54:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap4 2024-09-14T22:54:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:54:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:54:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:54:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:54:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:54:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap4-clean 2024-09-14T22:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-14T22:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:54:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap4-clean 2024-09-14T22:54:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:54:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:54:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T22:54:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:54:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:54:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 16776192 bytes 2024-09-14T22:54:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 43763712 bytes 2024-09-14T22:54:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 125646848 bytes 2024-09-14T22:54:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 154640384 bytes 2024-09-14T22:54:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 165428224 bytes 2024-09-14T22:54:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 245962752 bytes 2024-09-14T22:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 330795008 bytes 2024-09-14T22:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 380100608 bytes 2024-09-14T22:55:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 392644608 bytes 2024-09-14T22:55:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 426072064 bytes 2024-09-14T22:55:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 469545984 bytes 2024-09-14T22:55:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 535612416 bytes 2024-09-14T22:55:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 584932352 bytes 2024-09-14T22:55:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 615346176 bytes 2024-09-14T22:55:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 701041664 bytes 2024-09-14T22:55:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 736930816 bytes 2024-09-14T22:55:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 830268416 bytes 2024-09-14T22:55:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 876544000 bytes 2024-09-14T22:55:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 935822336 bytes 2024-09-14T22:55:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 960224256 bytes 2024-09-14T22:55:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 972145664 bytes 2024-09-14T22:55:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1005090816 bytes 2024-09-14T22:55:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1059867648 bytes 2024-09-14T22:55:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1115298816 bytes 2024-09-14T22:55:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1126760448 bytes 2024-09-14T22:55:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1171168256 bytes 2024-09-14T22:55:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1205440512 bytes 2024-09-14T22:55:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1264685056 bytes 2024-09-14T22:55:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1306438656 bytes 2024-09-14T22:55:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1348652032 bytes 2024-09-14T22:55:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1394390016 bytes 2024-09-14T22:55:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1452367872 bytes 2024-09-14T22:55:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1491103744 bytes 2024-09-14T22:55:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1540265984 bytes 2024-09-14T22:55:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1576954880 bytes 2024-09-14T22:55:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1580761088 bytes 2024-09-14T22:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1624214528 bytes 2024-09-14T22:55:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1675267072 bytes 2024-09-14T22:55:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1708498944 bytes 2024-09-14T22:55:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1756354560 bytes 2024-09-14T22:55:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1758059520 bytes 2024-09-14T22:55:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1780783104 bytes 2024-09-14T22:55:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1830241280 bytes 2024-09-14T22:55:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1935509504 bytes 2024-09-14T22:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1995528192 bytes 2024-09-14T22:55:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2098636800 bytes 2024-09-14T22:55:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2132595712 bytes 2024-09-14T22:55:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2197775360 bytes 2024-09-14T22:55:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2261038080 bytes 2024-09-14T22:55:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2273236992 bytes 2024-09-14T22:55:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2307166208 bytes 2024-09-14T22:55:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2341735424 bytes 2024-09-14T22:55:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2377156608 bytes 2024-09-14T22:55:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2426767360 bytes 2024-09-14T22:55:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2484536320 bytes 2024-09-14T22:55:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2521900032 bytes 2024-09-14T22:55:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2597392384 bytes 2024-09-14T22:55:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2644137984 bytes 2024-09-14T22:55:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2687117312 bytes 2024-09-14T22:55:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2725506048 bytes 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:55:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:55:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:55:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:55:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:55:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:55:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:55:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:55:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:55:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354410,"remote_snapshot_timestamp":1726354410,"replay_state":"idle"} 2024-09-14T22:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.1 on smithi089 2024-09-14T22:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:54:41 2024-09-14T22:55:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:55:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:55:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+stopped 2024-09-14T22:55:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: local image is primary 2024-09-14T22:55:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:54:38' 2024-09-14T22:55:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:55:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:55:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:55:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354410,"remote_snapshot_timestamp":1726354410,"replay_state":"idle"} 2024-09-14T22:55:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:55:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:55:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:55:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:55:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:55:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.1 on smithi089 2024-09-14T22:55:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:55:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:54:41' 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:54:41' 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:54:41' 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:55:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:55:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='OKOKOK142124262mirror.1smithi08919.1.1-324-gecab7be5trueOK43224325mirror.2smithi08919.1.1-324-gecab7be5falseOK43874390mirror.3smithi08919.1.1-324-gecab7be5falseOK44294432mirror.0smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354410,"remote_snapshot_timestamp":1726354410,"replay_state":"idle"}42124262mirror.1smithi0892024-09-14 22:54:41cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38up+stoppedlocal image is primary2024-09-14 22:54:38' 2024-09-14T22:55:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:55:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:55:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:54:41' 2024-09-14T22:55:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:55:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:55:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:55:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:55:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:55:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap5 2024-09-14T22:55:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap5 2024-09-14T22:55:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:55:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap5 2024-09-14T22:55:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:55:08.876 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5446025295746535) > weight (0.25) 2024-09-14T22:55:08.877 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7727808843476367) > weight (0.25) 2024-09-14T22:55:08.877 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3622831190035971) > weight (0.25) 2024-09-14T22:55:08.877 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6499637852893123) > weight (0.25) 2024-09-14T22:55:08.877 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-09-14T22:55:30.883 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4392887352008934) > weight (0.25) 2024-09-14T22:55:30.883 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6717534269343576) > weight (0.25) 2024-09-14T22:55:30.883 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4160381522963019) > weight (0.25) 2024-09-14T22:55:30.883 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.968448474840519) > weight (0.25) 2024-09-14T22:55:30.883 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-09-14T22:55:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:55:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:55:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:55:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:55:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:55:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:55:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:55:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:55:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:55:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:55:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:55:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:55:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:55:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:55:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:55:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:55:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:55:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:55:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:55:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:55:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:55:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:55:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:55:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:55:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:55:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:55:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:55:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:55:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:55:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:55:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:55:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:55:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:55:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:55:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:55:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:55:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:55:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:55:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:55:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:55:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:55:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:55:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:55:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:55:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:55:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:55:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:55:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:55:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:55:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:55:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:55:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:55:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:55:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:55:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:55:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:55:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:55:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 17 2024-09-14T22:55:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:55:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:55:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:55:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:55:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:55:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:55:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=17 2024-09-14T22:55:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:55:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:55:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:55:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:55:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:55:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:55:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:55:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:55:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:55:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:55:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:55:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:55:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:55:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 17 = '' 2024-09-14T22:55:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:55:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:55:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:55:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:55:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:55:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:55:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:55:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:55:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:55:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:55:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:55:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:55:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 17 = '' 2024-09-14T22:55:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:55:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:55:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:55:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:55:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:55:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:55:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:55:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:55:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:55:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:55:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:55:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:55:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 17 = '' 2024-09-14T22:55:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:55:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:55:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:55:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:55:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:55:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:55:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=17 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 17 = 17 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap5 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:55:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:55:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 10 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:54:55 2024 2024-09-14T22:55:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:55:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:55:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:55:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:55:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:55:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snap4-clean ']' 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap4-clean 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:55:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:55:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:55:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap5 2024-09-14T22:55:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap5 2024-09-14T22:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:55:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:55:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:55:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:55:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap5-clean 2024-09-14T22:55:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-14T22:55:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:55:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap5-clean 2024-09-14T22:55:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:55:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:55:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:55:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:55:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:55:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=5 2024-09-14T22:55:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:55:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:55:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 81609728 bytes 2024-09-14T22:55:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 113324032 bytes 2024-09-14T22:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 145123328 bytes 2024-09-14T22:55:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 272408576 bytes 2024-09-14T22:55:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 298867712 bytes 2024-09-14T22:55:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 328054784 bytes 2024-09-14T22:55:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 355209216 bytes 2024-09-14T22:55:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 374094848 bytes 2024-09-14T22:55:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 422989824 bytes 2024-09-14T22:55:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 462426112 bytes 2024-09-14T22:55:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 472478720 bytes 2024-09-14T22:55:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 523394048 bytes 2024-09-14T22:55:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 608613376 bytes 2024-09-14T22:55:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 612410368 bytes 2024-09-14T22:55:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 635977728 bytes 2024-09-14T22:55:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 686283776 bytes 2024-09-14T22:55:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 701031424 bytes 2024-09-14T22:55:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 743063552 bytes 2024-09-14T22:55:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 770464768 bytes 2024-09-14T22:55:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 800803840 bytes 2024-09-14T22:55:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 902422528 bytes 2024-09-14T22:55:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 918802432 bytes 2024-09-14T22:55:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 965043200 bytes 2024-09-14T22:55:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1006368768 bytes 2024-09-14T22:55:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1053118464 bytes 2024-09-14T22:55:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1104776192 bytes 2024-09-14T22:55:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1115270144 bytes 2024-09-14T22:55:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1173452800 bytes 2024-09-14T22:55:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1195398144 bytes 2024-09-14T22:55:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1288773632 bytes 2024-09-14T22:55:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1353029632 bytes 2024-09-14T22:55:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1413508096 bytes 2024-09-14T22:55:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1471194112 bytes 2024-09-14T22:55:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1526294528 bytes 2024-09-14T22:55:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1562321920 bytes 2024-09-14T22:55:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1651672064 bytes 2024-09-14T22:55:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1664950272 bytes 2024-09-14T22:55:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1699859456 bytes 2024-09-14T22:55:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1717001216 bytes 2024-09-14T22:55:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1736410112 bytes 2024-09-14T22:55:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1764115456 bytes 2024-09-14T22:55:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1841832960 bytes 2024-09-14T22:55:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1865823232 bytes 2024-09-14T22:55:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1890603008 bytes 2024-09-14T22:55:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1950503936 bytes 2024-09-14T22:55:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1989339136 bytes 2024-09-14T22:55:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2065591296 bytes 2024-09-14T22:55:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2092578816 bytes 2024-09-14T22:55:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2141848576 bytes 2024-09-14T22:55:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:55:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:55:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:55:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:55:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:55:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:55:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:55:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:55:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:55:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:55:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:55:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:55:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:55:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:55:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:55:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:55:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:55:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:55:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:55:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":528570368.0,"last_snapshot_bytes":528570368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354489,"remote_snapshot_timestamp":1726354489,"replay_state":"idle"} 2024-09-14T22:55:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.1 on smithi089 2024-09-14T22:55:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:55:38 2024-09-14T22:55:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:55:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:55:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+stopped 2024-09-14T22:55:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: local image is primary 2024-09-14T22:55:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:55:38' 2024-09-14T22:55:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:55:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:55:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:55:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":528570368.0,"last_snapshot_bytes":528570368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354489,"remote_snapshot_timestamp":1726354489,"replay_state":"idle"} 2024-09-14T22:55:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:55:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:55:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:55:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:55:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:55:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.1 on smithi089 2024-09-14T22:55:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:55:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:55:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:55:38' 2024-09-14T22:55:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:55:38' 2024-09-14T22:55:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:55:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:55:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:55:38' 2024-09-14T22:55:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:55:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:55:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:55:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='OKOKOK142124262mirror.1smithi08919.1.1-324-gecab7be5trueOK43224325mirror.2smithi08919.1.1-324-gecab7be5falseOK43874390mirror.3smithi08919.1.1-324-gecab7be5falseOK44294432mirror.0smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":528570368.0,"last_snapshot_bytes":528570368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354489,"remote_snapshot_timestamp":1726354489,"replay_state":"idle"}42124262mirror.1smithi0892024-09-14 22:55:38cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38up+stoppedlocal image is primary2024-09-14 22:55:38' 2024-09-14T22:55:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:55:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:55:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:55:38' 2024-09-14T22:55:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:55:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:55:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:55:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:55:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:55:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:55:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap6 2024-09-14T22:55:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap6 2024-09-14T22:55:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:55:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:55:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:55:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap6 2024-09-14T22:55:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:56:18.878 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4603399579315207) > weight (0.25) 2024-09-14T22:56:18.878 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.300437807922622) > weight (0.25) 2024-09-14T22:56:18.878 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9921112212184677) > weight (0.25) 2024-09-14T22:56:18.878 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-14T22:56:18.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-14T22:56:18.879 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T22:56:18.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T22:56:18.878+0000 7f689898f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 36937) UID: 1000 2024-09-14T22:56:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:56:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:56:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:56:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:56:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:56:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:56:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:56:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:56:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:56:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:56:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:56:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:56:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:56:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:56:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:56:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:56:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:56:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:56:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:56:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:56:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:56:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:56:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:56:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:56:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:56:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:56:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:56:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:56:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:56:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:56:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:56:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:56:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:56:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:56:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:56:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:56:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:56:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:56:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:56:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:56:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:56:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:56:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:56:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:56:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:56:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:56:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:56:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:56:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:56:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:56:23.879 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-14T22:56:23.880 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:56:23.880 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-14T22:56:23.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-14T22:56:23.881 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-14T22:56:23.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-14T22:56:23.884 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-14T22:56:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 20 2024-09-14T22:56:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:56:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:56:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:56:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:56:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:56:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:56:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=20 2024-09-14T22:56:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:56:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:56:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:56:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:56:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:56:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:56:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:56:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:56:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:56:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:56:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:56:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:56:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:56:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 20 = '' 2024-09-14T22:56:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:56:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:56:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:56:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:56:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:56:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:56:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:56:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:56:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 20 = '' 2024-09-14T22:56:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:56:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:56:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:56:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:56:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:56:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:56:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:56:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:56:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:56:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:56:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:56:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:56:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 20 = '' 2024-09-14T22:56:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:56:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:56:26.407 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T22:56:26.405+0000 7f5fe0b61640 -1 reset not still connected to 0x559266ca11e0 2024-09-14T22:56:26.408 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T22:56:26.406+0000 7f5fe0b61640 -1 reset not still connected to 0x559266e6ed00 2024-09-14T22:56:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:56:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:56:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:56:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=20 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 20 = 20 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap6 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:56:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:56:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 12 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:54:55 2024 2024-09-14T22:56:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:56:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:56:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:56:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:56:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:56:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:56:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:56:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snap5-clean ']' 2024-09-14T22:56:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-14T22:56:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap5-clean 2024-09-14T22:56:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:56:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:56:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:56:29.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:56:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:56:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:56:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap6 2024-09-14T22:56:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap6 2024-09-14T22:56:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:56:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:56:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:56:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:56:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:56:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:56:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:56:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap6-clean 2024-09-14T22:56:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-14T22:56:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:56:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap6-clean 2024-09-14T22:56:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:56:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:56:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:56:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:56:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:56:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T22:56:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:56:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:56:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 40275968 bytes 2024-09-14T22:56:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 67053568 bytes 2024-09-14T22:56:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 116140032 bytes 2024-09-14T22:56:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 152298496 bytes 2024-09-14T22:56:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 152881152 bytes 2024-09-14T22:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 163411968 bytes 2024-09-14T22:56:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 183819264 bytes 2024-09-14T22:56:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 216368128 bytes 2024-09-14T22:56:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 234297344 bytes 2024-09-14T22:56:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 269881344 bytes 2024-09-14T22:56:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 271499264 bytes 2024-09-14T22:56:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 301748224 bytes 2024-09-14T22:56:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 352329728 bytes 2024-09-14T22:56:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 404916224 bytes 2024-09-14T22:56:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 435717120 bytes 2024-09-14T22:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 514634752 bytes 2024-09-14T22:56:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 622894080 bytes 2024-09-14T22:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 738405376 bytes 2024-09-14T22:56:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 786918400 bytes 2024-09-14T22:56:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 851881984 bytes 2024-09-14T22:56:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 886367232 bytes 2024-09-14T22:56:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 939660288 bytes 2024-09-14T22:56:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 962553856 bytes 2024-09-14T22:56:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1029543936 bytes 2024-09-14T22:56:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1087893504 bytes 2024-09-14T22:56:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1138691072 bytes 2024-09-14T22:56:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1166939136 bytes 2024-09-14T22:56:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1239400448 bytes 2024-09-14T22:56:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1246643200 bytes 2024-09-14T22:56:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1299750912 bytes 2024-09-14T22:56:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1380674560 bytes 2024-09-14T22:56:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1443539968 bytes 2024-09-14T22:56:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1452532736 bytes 2024-09-14T22:56:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1505343488 bytes 2024-09-14T22:56:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1523530752 bytes 2024-09-14T22:56:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1597795328 bytes 2024-09-14T22:56:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1623734272 bytes 2024-09-14T22:56:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1634107392 bytes 2024-09-14T22:56:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1703552000 bytes 2024-09-14T22:56:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1772219392 bytes 2024-09-14T22:56:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1791230976 bytes 2024-09-14T22:56:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1857922048 bytes 2024-09-14T22:56:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1887672320 bytes 2024-09-14T22:56:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1976421376 bytes 2024-09-14T22:56:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1979636736 bytes 2024-09-14T22:56:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1994672128 bytes 2024-09-14T22:56:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2011925504 bytes 2024-09-14T22:56:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2102470656 bytes 2024-09-14T22:56:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2134462464 bytes 2024-09-14T22:56:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2184668160 bytes 2024-09-14T22:56:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2188980224 bytes 2024-09-14T22:56:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2234421248 bytes 2024-09-14T22:56:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2256695296 bytes 2024-09-14T22:56:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2334140416 bytes 2024-09-14T22:56:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2379899904 bytes 2024-09-14T22:56:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2498118656 bytes 2024-09-14T22:56:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2525978624 bytes 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:56:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:56:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:56:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:56:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:56:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:56:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:56:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:56:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:56:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":17727249.07,"bytes_per_snapshot":534107648.0,"last_snapshot_bytes":531817472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354583,"remote_snapshot_timestamp":1726354583,"replay_state":"idle"} 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.1 on smithi089 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:56:38 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: down+unknown 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: status not found 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: ' 2024-09-14T22:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:56:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:56:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:56:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":17727249.07,"bytes_per_snapshot":534107648.0,"last_snapshot_bytes":531817472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354583,"remote_snapshot_timestamp":1726354583,"replay_state":"idle"} 2024-09-14T22:56:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:56:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:56:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:56:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:56:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:56:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.1 on smithi089 2024-09-14T22:56:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:56:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:56:38' 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:56:38' 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:56:38' 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:56:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:56:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='WARNINGOKWARNING142124262mirror.1smithi08919.1.1-324-gecab7be5trueOK43224325mirror.2smithi08919.1.1-324-gecab7be5falseOK43874390mirror.3smithi08919.1.1-324-gecab7be5falseOK44294432mirror.0smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":17727249.07,"bytes_per_snapshot":534107648.0,"last_snapshot_bytes":531817472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354583,"remote_snapshot_timestamp":1726354583,"replay_state":"idle"}42124262mirror.1smithi0892024-09-14 22:56:38cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38down+unknownstatus not found' 2024-09-14T22:56:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:56:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:56:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:56:38' 2024-09-14T22:56:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:56:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:56:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:56:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:56:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:56:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:56:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:56:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap7 2024-09-14T22:56:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap7 2024-09-14T22:56:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:56:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:56:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:56:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap7 2024-09-14T22:56:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:56:53.884 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5966068767501791) > weight (0.25) 2024-09-14T22:56:53.884 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7799933988819837) > weight (0.25) 2024-09-14T22:56:53.884 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7927847502325974) > weight (0.25) 2024-09-14T22:56:53.884 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-14T22:56:53.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-14T22:56:53.885 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-14T22:56:53.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T22:56:53.884+0000 7f8b07a52640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 39802) UID: 1000 2024-09-14T22:56:54.886 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-14T22:56:54.886 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:56:54.886 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-14T22:56:54.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T22:56:54.887 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T22:56:54.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T22:56:54.890 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:57:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:57:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:57:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:57:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:57:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:57:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:57:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:57:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:57:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:57:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:57:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:57:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:57:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:57:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:57:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:57:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:57:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:57:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:57:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:57:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:57:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:57:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:57:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:57:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:57:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:57:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:57:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:57:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:57:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:57:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:57:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:57:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:57:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 23 2024-09-14T22:57:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:57:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:57:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:57:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:57:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:57:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=23 2024-09-14T22:57:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:57:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:57:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:57:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:57:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:57:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:57:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:57:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:57:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:57:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:57:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:57:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 23 = '' 2024-09-14T22:57:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:57:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:57:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:57:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:57:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:57:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:57:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:57:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:57:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 23 = '' 2024-09-14T22:57:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:57:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:57:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:57:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:57:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:57:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:57:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:57:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:57:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:57:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:57:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 23 = '' 2024-09-14T22:57:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:57:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:57:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:57:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:57:15.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:57:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=23 2024-09-14T22:57:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 23 = 23 2024-09-14T22:57:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:57:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-14T22:57:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap7 2024-09-14T22:57:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:57:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:57:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:57:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 14 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:57:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:56:29 2024 2024-09-14T22:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:57:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:57:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:57:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snap6-clean ']' 2024-09-14T22:57:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-14T22:57:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap6-clean 2024-09-14T22:57:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:57:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:57:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:57:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:57:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:57:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:57:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap7 2024-09-14T22:57:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap7 2024-09-14T22:57:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:57:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:57:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:57:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:57:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:57:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap7-clean 2024-09-14T22:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-14T22:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap7-clean 2024-09-14T22:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:57:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:57:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:57:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:57:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=9 2024-09-14T22:57:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:57:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:57:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 78066688 bytes 2024-09-14T22:57:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 141166592 bytes 2024-09-14T22:57:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 164252672 bytes 2024-09-14T22:57:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 179919872 bytes 2024-09-14T22:57:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 250147840 bytes 2024-09-14T22:57:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 276078592 bytes 2024-09-14T22:57:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 328119296 bytes 2024-09-14T22:57:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 350902272 bytes 2024-09-14T22:57:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 401551360 bytes 2024-09-14T22:57:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 440233984 bytes 2024-09-14T22:57:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 470028288 bytes 2024-09-14T22:57:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 527949824 bytes 2024-09-14T22:57:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 552445952 bytes 2024-09-14T22:57:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 584856576 bytes 2024-09-14T22:57:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 616336384 bytes 2024-09-14T22:57:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 639755264 bytes 2024-09-14T22:57:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 659208192 bytes 2024-09-14T22:57:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 675272704 bytes 2024-09-14T22:57:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 725831680 bytes 2024-09-14T22:57:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 774117376 bytes 2024-09-14T22:57:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 793528320 bytes 2024-09-14T22:57:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 809326592 bytes 2024-09-14T22:57:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 867715072 bytes 2024-09-14T22:57:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 922586112 bytes 2024-09-14T22:57:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 946246656 bytes 2024-09-14T22:57:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 994843648 bytes 2024-09-14T22:57:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1019149312 bytes 2024-09-14T22:57:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1019753472 bytes 2024-09-14T22:57:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1052182528 bytes 2024-09-14T22:57:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1130945536 bytes 2024-09-14T22:57:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1195957248 bytes 2024-09-14T22:57:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1253238784 bytes 2024-09-14T22:57:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1326953472 bytes 2024-09-14T22:57:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1351355392 bytes 2024-09-14T22:57:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1390649344 bytes 2024-09-14T22:57:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1391279104 bytes 2024-09-14T22:57:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1442257920 bytes 2024-09-14T22:57:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1481275392 bytes 2024-09-14T22:57:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1569402880 bytes 2024-09-14T22:57:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1598740480 bytes 2024-09-14T22:57:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1624790016 bytes 2024-09-14T22:57:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1656740864 bytes 2024-09-14T22:57:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1741625344 bytes 2024-09-14T22:57:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1792434176 bytes 2024-09-14T22:57:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1811334144 bytes 2024-09-14T22:57:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1850937344 bytes 2024-09-14T22:57:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1905412096 bytes 2024-09-14T22:57:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1929851904 bytes 2024-09-14T22:57:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1977361408 bytes 2024-09-14T22:57:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2013466624 bytes 2024-09-14T22:57:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2043765760 bytes 2024-09-14T22:57:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2086857728 bytes 2024-09-14T22:57:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2206305280 bytes 2024-09-14T22:57:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2239393792 bytes 2024-09-14T22:57:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2247114752 bytes 2024-09-14T22:57:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2272813056 bytes 2024-09-14T22:57:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2306139136 bytes 2024-09-14T22:57:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2388162560 bytes 2024-09-14T22:57:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2388759552 bytes 2024-09-14T22:57:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2389361664 bytes 2024-09-14T22:57:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2397280256 bytes 2024-09-14T22:57:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2457480192 bytes 2024-09-14T22:57:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2488294400 bytes 2024-09-14T22:57:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2532486144 bytes 2024-09-14T22:57:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2560777216 bytes 2024-09-14T22:57:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2566760448 bytes 2024-09-14T22:57:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2567481344 bytes 2024-09-14T22:57:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2634195968 bytes 2024-09-14T22:57:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2660830208 bytes 2024-09-14T22:57:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2714256384 bytes 2024-09-14T22:57:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2746857472 bytes 2024-09-14T22:57:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2794329088 bytes 2024-09-14T22:57:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2827014144 bytes 2024-09-14T22:57:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2875723776 bytes 2024-09-14T22:57:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 2928493568 bytes 2024-09-14T22:57:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 2974723072 bytes 2024-09-14T22:57:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3049498624 bytes 2024-09-14T22:57:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3095944192 bytes 2024-09-14T22:57:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3141872640 bytes 2024-09-14T22:57:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3165075456 bytes 2024-09-14T22:57:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3248856064 bytes 2024-09-14T22:57:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3249526784 bytes 2024-09-14T22:57:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3272174592 bytes 2024-09-14T22:57:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3293259776 bytes 2024-09-14T22:57:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3311831040 bytes 2024-09-14T22:57:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3345022976 bytes 2024-09-14T22:57:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3421897728 bytes 2024-09-14T22:57:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3422507008 bytes 2024-09-14T22:57:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3447983104 bytes 2024-09-14T22:57:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3499311104 bytes 2024-09-14T22:57:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3510907904 bytes 2024-09-14T22:57:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:57:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:57:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:57:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:57:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:57:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:57:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:57:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:57:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:57:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:57:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:57:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:57:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534107648.0,"last_snapshot_bytes":531817472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354583,"remote_snapshot_timestamp":1726354583,"replay_state":"idle"} 2024-09-14T22:57:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.1 on smithi089 2024-09-14T22:57:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:57:08 2024-09-14T22:57:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:57:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:57:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+stopped 2024-09-14T22:57:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: local image is primary 2024-09-14T22:57:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:57:08' 2024-09-14T22:57:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:57:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:57:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:57:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534107648.0,"last_snapshot_bytes":531817472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354583,"remote_snapshot_timestamp":1726354583,"replay_state":"idle"} 2024-09-14T22:57:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:57:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:57:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:57:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:57:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:57:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.1 on smithi089 2024-09-14T22:57:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:57:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:57:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:57:08' 2024-09-14T22:57:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:57:08' 2024-09-14T22:57:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:57:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:57:08' 2024-09-14T22:57:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:57:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:57:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:57:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='OKOKOK142124262mirror.1smithi08919.1.1-324-gecab7be5trueOK43224325mirror.2smithi08919.1.1-324-gecab7be5falseOK44294432mirror.0smithi08919.1.1-324-gecab7be5falseOK45264529mirror.3smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534107648.0,"last_snapshot_bytes":531817472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354583,"remote_snapshot_timestamp":1726354583,"replay_state":"idle"}42124262mirror.1smithi0892024-09-14 22:57:08cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38up+stoppedlocal image is primary2024-09-14 22:57:08' 2024-09-14T22:57:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:57:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:57:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:57:08' 2024-09-14T22:57:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:57:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:57:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:57:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:57:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:57:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:57:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:57:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap8 2024-09-14T22:57:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap8 2024-09-14T22:57:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap8 2024-09-14T22:57:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:57:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:57:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:57:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:57:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:57:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:57:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:57:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:57:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:57:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:57:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:57:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:57:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:57:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:57:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:57:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:57:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:57:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:57:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:57:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:57:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:57:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:57:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:57:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:57:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:57:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:57:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:57:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:57:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:57:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:57:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:57:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:57:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:57:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:57:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:57:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:57:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:57:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:57:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:57:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:57:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:57:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:57:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:57:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:57:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:57:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:57:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:57:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:57:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:57:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:57:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:57:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:57:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:57:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:57:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:57:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:57:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:58:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 26 2024-09-14T22:58:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:58:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:58:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:58:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:58:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:58:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=26 2024-09-14T22:58:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:58:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:58:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:58:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:58:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:58:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:58:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:58:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:58:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:58:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:58:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 26 = '' 2024-09-14T22:58:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:58:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:58:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:58:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:58:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:58:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:58:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:58:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:58:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:58:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 26 = '' 2024-09-14T22:58:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:58:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:58:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:58:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:58:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 26 = '' 2024-09-14T22:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:58:02.884 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T22:58:02.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T22:58:02.885 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-14T22:58:02.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T22:58:02.885+0000 7f0bc09c8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 36850) UID: 1000 2024-09-14T22:58:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:58:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:58:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=26 2024-09-14T22:58:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 26 = 26 2024-09-14T22:58:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-14T22:58:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap8 2024-09-14T22:58:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:58:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:58:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:58:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-14T22:58:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:58:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:58:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:58:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 16 2024-09-14T22:58:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:58:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:58:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:56:29 2024 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snap7-clean ']' 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap7-clean 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:58:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:58:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:58:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap8 2024-09-14T22:58:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap8 2024-09-14T22:58:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:58:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:58:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:58:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:58:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:58:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap8-clean 2024-09-14T22:58:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-14T22:58:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:58:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap8-clean 2024-09-14T22:58:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:58:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:58:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:58:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:58:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:58:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=13 2024-09-14T22:58:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:58:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:58:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 83577856 bytes 2024-09-14T22:58:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 119446528 bytes 2024-09-14T22:58:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 161518592 bytes 2024-09-14T22:58:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 214539264 bytes 2024-09-14T22:58:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 233714688 bytes 2024-09-14T22:58:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 287578112 bytes 2024-09-14T22:58:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 310853632 bytes 2024-09-14T22:58:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 374560768 bytes 2024-09-14T22:58:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 391043072 bytes 2024-09-14T22:58:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 414539776 bytes 2024-09-14T22:58:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 463384576 bytes 2024-09-14T22:58:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 495416320 bytes 2024-09-14T22:58:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 547888128 bytes 2024-09-14T22:58:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 595307520 bytes 2024-09-14T22:58:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 700875776 bytes 2024-09-14T22:58:10.886 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T22:58:10.887 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:58:10.887 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T22:58:10.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T22:58:10.887 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T22:58:10.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T22:58:10.891 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-14T22:58:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 796504064 bytes 2024-09-14T22:58:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 833182720 bytes 2024-09-14T22:58:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 878731264 bytes 2024-09-14T22:58:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 960896000 bytes 2024-09-14T22:58:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 1012515840 bytes 2024-09-14T22:58:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 1042900992 bytes 2024-09-14T22:58:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1091728384 bytes 2024-09-14T22:58:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1163512832 bytes 2024-09-14T22:58:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1192127488 bytes 2024-09-14T22:58:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1216771072 bytes 2024-09-14T22:58:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1275146240 bytes 2024-09-14T22:58:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1302301696 bytes 2024-09-14T22:58:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1343623168 bytes 2024-09-14T22:58:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1415641088 bytes 2024-09-14T22:58:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1447023616 bytes 2024-09-14T22:58:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1467943936 bytes 2024-09-14T22:58:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1468557312 bytes 2024-09-14T22:58:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1548337152 bytes 2024-09-14T22:58:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1600865280 bytes 2024-09-14T22:58:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1650274304 bytes 2024-09-14T22:58:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1674486784 bytes 2024-09-14T22:58:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1689318400 bytes 2024-09-14T22:58:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1710804992 bytes 2024-09-14T22:58:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1794828288 bytes 2024-09-14T22:58:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1822065664 bytes 2024-09-14T22:58:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1944922112 bytes 2024-09-14T22:58:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 2013537280 bytes 2024-09-14T22:58:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 2102605824 bytes 2024-09-14T22:58:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 2130435072 bytes 2024-09-14T22:58:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2196772864 bytes 2024-09-14T22:58:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2265839616 bytes 2024-09-14T22:58:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2310020096 bytes 2024-09-14T22:58:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2366440448 bytes 2024-09-14T22:58:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2465664000 bytes 2024-09-14T22:58:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2501115904 bytes 2024-09-14T22:58:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2541639680 bytes 2024-09-14T22:58:14.890 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8733605387675487) > weight (0.25) 2024-09-14T22:58:14.890 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4386282682026531) > weight (0.25) 2024-09-14T22:58:14.891 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7289290629865481) > weight (0.25) 2024-09-14T22:58:14.891 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.873873180505368) > weight (0.25) 2024-09-14T22:58:14.891 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-09-14T22:58:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2651546624 bytes 2024-09-14T22:58:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2673425408 bytes 2024-09-14T22:58:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2730470400 bytes 2024-09-14T22:58:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2731088896 bytes 2024-09-14T22:58:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2835147776 bytes 2024-09-14T22:58:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2866000896 bytes 2024-09-14T22:58:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2910486528 bytes 2024-09-14T22:58:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2966530048 bytes 2024-09-14T22:58:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2989963264 bytes 2024-09-14T22:58:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 3031368704 bytes 2024-09-14T22:58:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 3062566912 bytes 2024-09-14T22:58:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 3103304704 bytes 2024-09-14T22:58:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 3145167872 bytes 2024-09-14T22:58:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 3190916096 bytes 2024-09-14T22:58:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 3250155520 bytes 2024-09-14T22:58:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 3334982656 bytes 2024-09-14T22:58:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 3369829376 bytes 2024-09-14T22:58:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3410046976 bytes 2024-09-14T22:58:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3474409472 bytes 2024-09-14T22:58:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3521498112 bytes 2024-09-14T22:58:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3529802752 bytes 2024-09-14T22:58:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3591514112 bytes 2024-09-14T22:58:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3594594304 bytes 2024-09-14T22:58:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3651762176 bytes 2024-09-14T22:58:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3723001856 bytes 2024-09-14T22:58:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3755031552 bytes 2024-09-14T22:58:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3772804096 bytes 2024-09-14T22:58:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3810746368 bytes 2024-09-14T22:58:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3924426752 bytes 2024-09-14T22:58:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 4043052032 bytes 2024-09-14T22:58:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 4057649152 bytes 2024-09-14T22:58:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 4058280960 bytes 2024-09-14T22:58:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 4109972480 bytes 2024-09-14T22:58:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 4110573568 bytes 2024-09-14T22:58:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 4128522240 bytes 2024-09-14T22:58:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 4189775872 bytes 2024-09-14T22:58:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 4226919424 bytes 2024-09-14T22:58:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 4288223232 bytes 2024-09-14T22:58:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 4318322688 bytes 2024-09-14T22:58:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 4388846592 bytes 2024-09-14T22:58:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 4425593856 bytes 2024-09-14T22:58:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 4435375104 bytes 2024-09-14T22:58:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4464350208 bytes 2024-09-14T22:58:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4524498944 bytes 2024-09-14T22:58:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4550719488 bytes 2024-09-14T22:58:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4551326720 bytes 2024-09-14T22:58:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4570963968 bytes 2024-09-14T22:58:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4571583488 bytes 2024-09-14T22:58:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4657174528 bytes 2024-09-14T22:58:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4668603392 bytes 2024-09-14T22:58:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4722316288 bytes 2024-09-14T22:58:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4754832384 bytes 2024-09-14T22:58:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4813775872 bytes 2024-09-14T22:58:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4892862464 bytes 2024-09-14T22:58:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4941689856 bytes 2024-09-14T22:58:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4996592640 bytes 2024-09-14T22:58:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 5058738176 bytes 2024-09-14T22:58:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 5110106112 bytes 2024-09-14T22:58:21.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11000 IOs, 5163846656 bytes 2024-09-14T22:58:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11100 IOs, 5190464512 bytes 2024-09-14T22:58:21.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11200 IOs, 5279107072 bytes 2024-09-14T22:58:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11300 IOs, 5341837312 bytes 2024-09-14T22:58:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11400 IOs, 5374023680 bytes 2024-09-14T22:58:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11500 IOs, 5409763328 bytes 2024-09-14T22:58:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11600 IOs, 5457865728 bytes 2024-09-14T22:58:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11700 IOs, 5556379648 bytes 2024-09-14T22:58:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:58:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:58:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:58:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:58:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:58:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:58:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:58:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:58:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:58:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:58:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:58:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":17852040.53,"bytes_per_snapshot":533110272.0,"last_snapshot_bytes":535561216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354679,"remote_snapshot_timestamp":1726354679,"replay_state":"idle"} 2024-09-14T22:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.1 on smithi089 2024-09-14T22:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:58:08 2024-09-14T22:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+stopped 2024-09-14T22:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: local image is primary 2024-09-14T22:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:58:08' 2024-09-14T22:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:58:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:58:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:58:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":17852040.53,"bytes_per_snapshot":533110272.0,"last_snapshot_bytes":535561216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354679,"remote_snapshot_timestamp":1726354679,"replay_state":"idle"} 2024-09-14T22:58:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:58:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:58:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:58:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:58:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:58:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.1 on smithi089 2024-09-14T22:58:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:58:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:58:08' 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:58:08' 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:58:08' 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:58:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:58:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='OKOKOK142124262mirror.1smithi08919.1.1-324-gecab7be5trueOK43224325mirror.2smithi08919.1.1-324-gecab7be5falseOK44294432mirror.0smithi08919.1.1-324-gecab7be5falseOK45264529mirror.3smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":17852040.53,"bytes_per_snapshot":533110272.0,"last_snapshot_bytes":535561216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354679,"remote_snapshot_timestamp":1726354679,"replay_state":"idle"}42124262mirror.1smithi0892024-09-14 22:58:08cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38up+stoppedlocal image is primary2024-09-14 22:58:08' 2024-09-14T22:58:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:58:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:58:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:58:08' 2024-09-14T22:58:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:58:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:58:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:58:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:58:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:58:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap9 2024-09-14T22:58:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap9 2024-09-14T22:58:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:58:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap9 2024-09-14T22:58:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:58:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:58:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:58:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:58:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:58:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:58:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:58:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:58:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:58:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:58:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:58:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:58:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:58:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:58:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:58:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:58:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:58:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:58:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:58:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:58:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:58:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:58:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:58:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:58:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:58:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:58:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:58:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:58:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:58:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:58:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:58:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:58:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:58:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:58:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:58:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:58:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:58:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:58:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:58:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:58:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:58:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:58:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:58:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:58:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:58:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:58:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:58:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 29 2024-09-14T22:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:58:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:58:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:58:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=29 2024-09-14T22:58:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:58:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:58:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:58:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:58:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:58:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:58:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:58:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:58:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 29 = '' 2024-09-14T22:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:58:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:58:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:58:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:58:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:58:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:58:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:58:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:58:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 29 = '' 2024-09-14T22:58:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:58:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:58:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:58:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:58:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:58:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:58:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:58:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 29 = '' 2024-09-14T22:58:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:58:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:58:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:58:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:58:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:58:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=29 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 29 = 29 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap9 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:58:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:58:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-14T22:58:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 18 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:58:05 2024 2024-09-14T22:58:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:58:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:58:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:58:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:58:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:58:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snap8-clean ']' 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap8-clean 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:58:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:58:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:58:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap9 2024-09-14T22:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap9 2024-09-14T22:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap9-clean 2024-09-14T22:59:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-14T22:59:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap9-clean 2024-09-14T22:59:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:59:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:59:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T22:59:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T22:59:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T22:59:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T22:59:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T22:59:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 69970944 bytes 2024-09-14T22:59:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 178567168 bytes 2024-09-14T22:59:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 232044544 bytes 2024-09-14T22:59:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 232660992 bytes 2024-09-14T22:59:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 272327680 bytes 2024-09-14T22:59:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 341894144 bytes 2024-09-14T22:59:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 354072576 bytes 2024-09-14T22:59:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 437216256 bytes 2024-09-14T22:59:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 494516224 bytes 2024-09-14T22:59:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 525155328 bytes 2024-09-14T22:59:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 607088640 bytes 2024-09-14T22:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 663258112 bytes 2024-09-14T22:59:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 699202560 bytes 2024-09-14T22:59:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 772257792 bytes 2024-09-14T22:59:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 820355072 bytes 2024-09-14T22:59:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 835325952 bytes 2024-09-14T22:59:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 853428224 bytes 2024-09-14T22:59:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 894030848 bytes 2024-09-14T22:59:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 919990272 bytes 2024-09-14T22:59:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 973257728 bytes 2024-09-14T22:59:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 988950528 bytes 2024-09-14T22:59:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1023621120 bytes 2024-09-14T22:59:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1075128320 bytes 2024-09-14T22:59:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1114097664 bytes 2024-09-14T22:59:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1169452032 bytes 2024-09-14T22:59:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1185811456 bytes 2024-09-14T22:59:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1230313472 bytes 2024-09-14T22:59:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1296423936 bytes 2024-09-14T22:59:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1305414656 bytes 2024-09-14T22:59:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1385088000 bytes 2024-09-14T22:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1424465920 bytes 2024-09-14T22:59:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1475006464 bytes 2024-09-14T22:59:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1508520960 bytes 2024-09-14T22:59:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1549935616 bytes 2024-09-14T22:59:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1568541696 bytes 2024-09-14T22:59:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1593832448 bytes 2024-09-14T22:59:06.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1631320064 bytes 2024-09-14T22:59:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1672854528 bytes 2024-09-14T22:59:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1742604288 bytes 2024-09-14T22:59:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1776516096 bytes 2024-09-14T22:59:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1837647872 bytes 2024-09-14T22:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1897165824 bytes 2024-09-14T22:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1897782272 bytes 2024-09-14T22:59:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1911688192 bytes 2024-09-14T22:59:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1942641664 bytes 2024-09-14T22:59:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1984265216 bytes 2024-09-14T22:59:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2009796608 bytes 2024-09-14T22:59:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2057049088 bytes 2024-09-14T22:59:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2086878208 bytes 2024-09-14T22:59:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2164148224 bytes 2024-09-14T22:59:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2185674752 bytes 2024-09-14T22:59:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2246184960 bytes 2024-09-14T22:59:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2287107072 bytes 2024-09-14T22:59:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2335157248 bytes 2024-09-14T22:59:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2335755264 bytes 2024-09-14T22:59:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2380263424 bytes 2024-09-14T22:59:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T22:59:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T22:59:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T22:59:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T22:59:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:59:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:59:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:59:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:59:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:59:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T22:59:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:59:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local description_pattern= 2024-09-14T22:59:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local service_pattern= 2024-09-14T22:59:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:59:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:59:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T22:59:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='test: 2024-09-14T22:59:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: global_id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:59:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+replaying 2024-09-14T22:59:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: replaying, {"bytes_per_second":17891054.93,"bytes_per_snapshot":536146432.0,"last_snapshot_bytes":536731648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354733,"remote_snapshot_timestamp":1726354733,"replay_state":"idle"} 2024-09-14T22:59:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: service: mirror.1 on smithi089 2024-09-14T22:59:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:59:08 2024-09-14T22:59:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: peer_sites: 2024-09-14T22:59:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: name: cluster2 2024-09-14T22:59:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: state: up+stopped 2024-09-14T22:59:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: description: local image is primary 2024-09-14T22:59:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: last_update: 2024-09-14 22:59:08' 2024-09-14T22:59:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T22:59:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: state: up+replaying 2024-09-14T22:59:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^ description: .*' 2024-09-14T22:59:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: description: replaying, {"bytes_per_second":17891054.93,"bytes_per_snapshot":536146432.0,"last_snapshot_bytes":536731648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354733,"remote_snapshot_timestamp":1726354733,"replay_state":"idle"} 2024-09-14T22:59:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T22:59:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:59:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep '^up+' 2024-09-14T22:59:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:59:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T22:59:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: service: mirror.1 on smithi089 2024-09-14T22:59:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update 2024-09-14T22:59:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T22:59:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:59:08' 2024-09-14T22:59:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 22:59:08' 2024-09-14T22:59:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state_pattern=up+replaying 2024-09-14T22:59:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local 'prev_last_update=2024-09-14 22:59:08' 2024-09-14T22:59:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status 2024-09-14T22:59:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ CEPH_ARGS= 2024-09-14T22:59:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T22:59:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status='OKOKOK142124262mirror.1smithi08919.1.1-324-gecab7be5trueOK43224325mirror.2smithi08919.1.1-324-gecab7be5falseOK44294432mirror.0smithi08919.1.1-324-gecab7be5falseOK45264529mirror.3smithi08919.1.1-324-gecab7be5falseOKtest5fee3a67-30ec-492b-a757-4e423c6c23e3up+replayingreplaying, {"bytes_per_second":17891054.93,"bytes_per_snapshot":536146432.0,"last_snapshot_bytes":536731648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726354733,"remote_snapshot_timestamp":1726354733,"replay_state":"idle"}42124262mirror.1smithi0892024-09-14 22:59:08cluster2a4ac212d-4b54-4fda-99c8-ab004b275d38up+stoppedlocal image is primary2024-09-14 22:59:08' 2024-09-14T22:59:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_update state 2024-09-14T22:59:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T22:59:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ last_update='2024-09-14 22:59:08' 2024-09-14T22:59:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T22:59:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=up+replaying 2024-09-14T22:59:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo up+replaying 2024-09-14T22:59:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep up+replaying 2024-09-14T22:59:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:up+replaying 2024-09-14T22:59:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap10 2024-09-14T22:59:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap10 2024-09-14T22:59:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap10 2024-09-14T22:59:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-14T22:59:28.891 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6449770438331365) > weight (0.25) 2024-09-14T22:59:28.892 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5748271695271214) > weight (0.25) 2024-09-14T22:59:28.892 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5175380311781358) > weight (0.25) 2024-09-14T22:59:28.892 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-14T22:59:28.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-14T22:59:28.893 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-14T22:59:28.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T22:59:28.892+0000 7f4c3de12640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 41905) UID: 1000 2024-09-14T22:59:31.894 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-14T22:59:31.894 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:59:31.894 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-14T22:59:31.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T22:59:31.895 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T22:59:31.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T22:59:31.898 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-09-14T22:59:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:59:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:59:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:59:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:59:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:59:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:59:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:59:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:59:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:59:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:59:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:59:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:59:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:59:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:59:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:59:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:59:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:59:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:59:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:59:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:59:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:59:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:59:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:59:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:59:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:59:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:59:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:59:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:59:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:59:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:59:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:59:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:59:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:59:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:59:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:59:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:59:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:59:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:59:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:59:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:59:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:59:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:59:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:59:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:59:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:59:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:59:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:59:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:59:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:59:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:59:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:59:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:59:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:59:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:59:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:59:39.892 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3114705362338547) > weight (0.25) 2024-09-14T22:59:39.893 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-14T22:59:39.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-14T22:59:39.893 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-14T22:59:39.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T22:59:39.892+0000 7fee5d3cf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 40228) UID: 1000 2024-09-14T22:59:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 32 2024-09-14T22:59:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:59:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:59:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:59:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:59:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:59:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=32 2024-09-14T22:59:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:59:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:59:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:59:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:59:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:59:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:59:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:59:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:59:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:59:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:59:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:59:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 32 = '' 2024-09-14T22:59:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:59:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:59:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:59:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:59:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:59:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:59:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:59:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:59:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:59:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:59:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 32 = '' 2024-09-14T22:59:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:59:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T22:59:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:59:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:59:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:59:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:59:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:59:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:59:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:59:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:59:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 32 = '' 2024-09-14T22:59:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:59:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T22:59:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:59:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:59:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=32 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 32 = 32 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap10 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:59:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:59:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-14T22:59:45.894 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-14T22:59:45.894 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T22:59:45.894 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-14T22:59:45.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T22:59:45.895 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T22:59:45.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T22:59:45.897 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-14T22:59:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:59:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:59:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:59:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 20 2024-09-14T22:59:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:59:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:59:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:59:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:58:05 2024 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:59:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:59:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snap9-clean ']' 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap9-clean 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap10 2024-09-14T22:59:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap10 2024-09-14T22:59:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:59:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ clean_snap_name=snap10-clean 2024-09-14T22:59:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-14T22:59:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap10-clean 2024-09-14T22:59:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T22:59:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:59:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:59:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:59:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T22:59:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T22:59:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T22:59:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T22:59:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T22:59:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T22:59:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:59:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T22:59:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T22:59:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T22:59:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T22:59:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T22:59:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T22:59:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T22:59:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T22:59:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T22:59:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T22:59:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:59:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:59:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T22:59:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T22:59:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T22:59:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:59:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:59:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T22:59:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T22:59:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T22:59:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T22:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T22:59:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T22:59:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T22:59:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T22:59:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T22:59:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T22:59:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T22:59:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T22:59:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T22:59:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T22:59:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:59:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:59:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T22:59:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T22:59:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T22:59:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T22:59:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T22:59:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T22:59:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T22:59:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T22:59:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T22:59:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T22:59:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-test.status 2024-09-14T22:59:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:59:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:59:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T22:59:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-test.status 2024-09-14T22:59:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T22:59:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T22:59:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T22:59:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 34 2024-09-14T22:59:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:59:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T22:59:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.status 2024-09-14T22:59:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:59:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T22:59:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T22:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=34 2024-09-14T22:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:59:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T22:59:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:59:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:59:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T22:59:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:59:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T22:59:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T22:59:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T22:59:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T22:59:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 34 = '' 2024-09-14T22:59:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T22:59:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T22:59:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.status 2024-09-14T22:59:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T22:59:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T22:59:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T22:59:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=34 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 34 = 34 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap10-clean 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T22:59:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-14T22:59:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'test': 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 21 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a3d56baa93 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a3d56baa93 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 22:59:46 2024 2024-09-14T22:59:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 22:51:10 2024 2024-09-14T22:59:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T22:59:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T22:59:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T22:59:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5fee3a67-30ec-492b-a757-4e423c6c23e3 2024-09-14T22:59:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T22:59:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 10 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap1 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap1 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap1 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap1-clean 2024-09-14T22:59:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-14T22:59:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap1-clean 2024-09-14T22:59:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:59:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T22:59:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap2 2024-09-14T22:59:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap2 2024-09-14T22:59:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap2 2024-09-14T22:59:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:59:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap2-clean 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap2-clean 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T22:59:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T22:59:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T22:59:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap3 2024-09-14T23:00:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap3 2024-09-14T23:00:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap3 2024-09-14T23:00:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap3-clean 2024-09-14T23:00:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-14T23:00:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap3-clean 2024-09-14T23:00:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap4 2024-09-14T23:00:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap4 2024-09-14T23:00:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap4 2024-09-14T23:00:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap4-clean 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap4-clean 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap5 2024-09-14T23:00:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap5 2024-09-14T23:00:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap5 2024-09-14T23:00:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap5-clean 2024-09-14T23:00:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-14T23:00:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap5-clean 2024-09-14T23:00:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap6 2024-09-14T23:00:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap6 2024-09-14T23:00:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap6 2024-09-14T23:00:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap6-clean 2024-09-14T23:00:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-14T23:00:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap6-clean 2024-09-14T23:00:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap7 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap7 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap7 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap7-clean 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap7-clean 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap8 2024-09-14T23:00:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap8 2024-09-14T23:00:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap8 2024-09-14T23:00:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap8-clean 2024-09-14T23:00:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-14T23:00:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap8-clean 2024-09-14T23:00:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap9 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap9 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap9 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap9-clean 2024-09-14T23:00:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-14T23:00:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap9-clean 2024-09-14T23:00:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap10 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap10 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap10 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap10-clean 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap10-clean 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export 2024-09-14T23:00:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T23:00:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-test.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-test.export 2024-09-14T23:00:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 10 2024-09-14T23:00:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap1 2024-09-14T23:00:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-14T23:00:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap1 2024-09-14T23:00:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-14T23:00:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap1-clean 2024-09-14T23:00:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-14T23:00:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap1-clean 2024-09-14T23:00:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-14T23:00:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap2 2024-09-14T23:00:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-14T23:00:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap2 2024-09-14T23:00:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-14T23:00:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap2-clean 2024-09-14T23:00:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-14T23:00:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap2-clean 2024-09-14T23:00:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-14T23:00:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap3 2024-09-14T23:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-14T23:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap3 2024-09-14T23:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-14T23:00:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap3-clean 2024-09-14T23:00:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-14T23:00:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap3-clean 2024-09-14T23:00:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-14T23:00:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap4 2024-09-14T23:00:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-14T23:00:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap4 2024-09-14T23:00:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-14T23:00:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap4-clean 2024-09-14T23:00:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-14T23:00:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap4-clean 2024-09-14T23:00:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-14T23:00:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap5 2024-09-14T23:00:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-14T23:00:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap5 2024-09-14T23:00:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-14T23:00:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap5-clean 2024-09-14T23:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-14T23:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap5-clean 2024-09-14T23:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-14T23:00:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap6 2024-09-14T23:00:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-14T23:00:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap6 2024-09-14T23:00:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-14T23:00:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap6-clean 2024-09-14T23:00:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-14T23:00:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap6-clean 2024-09-14T23:00:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-14T23:00:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap7 2024-09-14T23:00:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-14T23:00:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap7 2024-09-14T23:00:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-14T23:00:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap7-clean 2024-09-14T23:00:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-14T23:00:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap7-clean 2024-09-14T23:00:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-14T23:00:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap8 2024-09-14T23:00:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-14T23:00:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap8 2024-09-14T23:00:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-14T23:00:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap8-clean 2024-09-14T23:00:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-14T23:00:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap8-clean 2024-09-14T23:00:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-14T23:00:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap9 2024-09-14T23:00:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-14T23:00:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap9 2024-09-14T23:00:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-14T23:00:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap9-clean 2024-09-14T23:00:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-14T23:00:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap9-clean 2024-09-14T23:00:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-14T23:00:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:00:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap10 2024-09-14T23:00:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-14T23:00:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap10 2024-09-14T23:00:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-14T23:00:43.899 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T23:00:43.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T23:00:43.900 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-14T23:00:43.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:00:43.900+0000 7f2e17185640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 40468) UID: 1000 2024-09-14T23:00:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap10-clean 2024-09-14T23:00:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-14T23:00:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap10-clean 2024-09-14T23:00:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-14T23:00:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:00:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror test 2024-09-14T23:00:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:00:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:00:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror test 2024-09-14T23:00:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-14T23:00:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-14T23:00:45.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:00:45.714+0000 7fe8dd1c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c1de3acea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:00:45.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:00:45.714+0000 7fe8e39d5640 -1 rbd::mirror::ImageReplayer: 0x55c1de6b0500 [3/5fee3a67-30ec-492b-a757-4e423c6c23e3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:00:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-14T23:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:00:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror test 2024-09-14T23:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=test 2024-09-14T23:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-14T23:00:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:00:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-14T23:00:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:00:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:00:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:00:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=test 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror test 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=test 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-14T23:00:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:00:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-14T23:00:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:00:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:00:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:00:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ testlog 'TEST: create many images' 2024-09-14T23:00:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ tee -a /tmp/tmp.OxLGv2e3xU/rbd-mirror.test.log 2024-09-14T23:00:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ date '+%F %T' 2024-09-14T23:00:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo 2024-09-14 23:00:46 TEST: create many images 2024-09-14T23:00:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:2024-09-14 23:00:46 TEST: create many images 2024-09-14T23:00:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ snap_name=snap 2024-09-14T23:00:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 50 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_1 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:00:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:00:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:00:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2024-09-14T23:00:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:00:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:00:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-14T23:00:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:00:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:00:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-14T23:00:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:00:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-14T23:00:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_1': 2024-09-14T23:00:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:00:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:00:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:00:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12775d902795 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12775d902795 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:00:46 2024 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:00:46 2024 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:00:46 2024 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 23baaf62-5401-42f0-ae98-c17bcc416c67 2024-09-14T23:00:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:00:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:00:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-14T23:00:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:00:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:00:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=8 2024-09-14T23:00:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:00:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:00:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 63579136 bytes 2024-09-14T23:00:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 83446784 bytes 2024-09-14T23:00:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 123399168 bytes 2024-09-14T23:00:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 168910848 bytes 2024-09-14T23:00:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 202344448 bytes 2024-09-14T23:00:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 275054592 bytes 2024-09-14T23:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 297577472 bytes 2024-09-14T23:00:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 389504000 bytes 2024-09-14T23:00:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 488844288 bytes 2024-09-14T23:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 554041344 bytes 2024-09-14T23:00:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 608899072 bytes 2024-09-14T23:00:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 650042368 bytes 2024-09-14T23:00:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 672747520 bytes 2024-09-14T23:00:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 756826112 bytes 2024-09-14T23:00:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 799509504 bytes 2024-09-14T23:00:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 868597760 bytes 2024-09-14T23:00:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 916423680 bytes 2024-09-14T23:00:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 931671040 bytes 2024-09-14T23:00:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 950306816 bytes 2024-09-14T23:00:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 963868672 bytes 2024-09-14T23:00:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 986967040 bytes 2024-09-14T23:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1020647424 bytes 2024-09-14T23:00:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1057106944 bytes 2024-09-14T23:00:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1071804416 bytes 2024-09-14T23:00:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1132251136 bytes 2024-09-14T23:00:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1132882944 bytes 2024-09-14T23:00:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1151768576 bytes 2024-09-14T23:00:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1181061120 bytes 2024-09-14T23:00:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1204973568 bytes 2024-09-14T23:00:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1225209856 bytes 2024-09-14T23:00:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1272908800 bytes 2024-09-14T23:00:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1303999488 bytes 2024-09-14T23:00:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1332506624 bytes 2024-09-14T23:00:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1378842624 bytes 2024-09-14T23:00:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1458113536 bytes 2024-09-14T23:00:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1487663104 bytes 2024-09-14T23:00:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1510808576 bytes 2024-09-14T23:00:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1566815232 bytes 2024-09-14T23:00:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1651248128 bytes 2024-09-14T23:00:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1702962176 bytes 2024-09-14T23:00:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1725341696 bytes 2024-09-14T23:00:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1817424896 bytes 2024-09-14T23:00:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1850710016 bytes 2024-09-14T23:00:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1898063872 bytes 2024-09-14T23:00:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1948576768 bytes 2024-09-14T23:00:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1972624384 bytes 2024-09-14T23:00:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2000337920 bytes 2024-09-14T23:00:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2059289600 bytes 2024-09-14T23:00:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2124877824 bytes 2024-09-14T23:00:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2167813120 bytes 2024-09-14T23:00:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2211211264 bytes 2024-09-14T23:00:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2232552448 bytes 2024-09-14T23:00:52.900 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T23:00:52.901 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:00:52.901 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T23:00:52.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T23:00:52.901 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T23:00:52.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T23:00:52.905 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-14T23:00:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2289811456 bytes 2024-09-14T23:00:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2336345088 bytes 2024-09-14T23:00:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2386101248 bytes 2024-09-14T23:00:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2398362624 bytes 2024-09-14T23:00:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2409786368 bytes 2024-09-14T23:00:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2455542784 bytes 2024-09-14T23:00:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2520757248 bytes 2024-09-14T23:00:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2594307072 bytes 2024-09-14T23:00:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2649029632 bytes 2024-09-14T23:00:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2719585280 bytes 2024-09-14T23:00:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2732537856 bytes 2024-09-14T23:00:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2752229376 bytes 2024-09-14T23:00:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2803813376 bytes 2024-09-14T23:00:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2804428800 bytes 2024-09-14T23:00:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2844839936 bytes 2024-09-14T23:00:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2878014464 bytes 2024-09-14T23:00:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2878623744 bytes 2024-09-14T23:00:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2969542656 bytes 2024-09-14T23:00:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2970151936 bytes 2024-09-14T23:00:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3051634688 bytes 2024-09-14T23:00:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3078754304 bytes 2024-09-14T23:00:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3117926400 bytes 2024-09-14T23:00:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3154462720 bytes 2024-09-14T23:00:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3155083264 bytes 2024-09-14T23:00:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3183434752 bytes 2024-09-14T23:00:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3203551232 bytes 2024-09-14T23:00:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3232844800 bytes 2024-09-14T23:00:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3234298880 bytes 2024-09-14T23:00:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:00:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:00:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:00:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:00:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_2 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:00:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-14T23:00:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:00:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:00:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:00:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:00:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:00:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2024-09-14T23:00:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:00:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:00:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:00:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-14T23:00:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:00:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:00:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-14T23:00:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:00:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_2': 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1287c22b0b97 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1287c22b0b97 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:00:55 2024 2024-09-14T23:00:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:00:55 2024 2024-09-14T23:00:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:00:55 2024 2024-09-14T23:00:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:00:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:00:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 13ef0e31-a893-45b8-a2c7-d21116cdc208 2024-09-14T23:00:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:00:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:00:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-14T23:00:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:00:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:00:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:00:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:00:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=7 2024-09-14T23:00:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:00:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:00:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 50843648 bytes 2024-09-14T23:00:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 118171648 bytes 2024-09-14T23:00:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 158286848 bytes 2024-09-14T23:00:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 237593600 bytes 2024-09-14T23:00:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 253704192 bytes 2024-09-14T23:00:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 294514688 bytes 2024-09-14T23:00:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 326815744 bytes 2024-09-14T23:00:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 347738112 bytes 2024-09-14T23:00:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 371171328 bytes 2024-09-14T23:00:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 400247808 bytes 2024-09-14T23:00:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 400867328 bytes 2024-09-14T23:00:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 401451008 bytes 2024-09-14T23:00:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 501805056 bytes 2024-09-14T23:00:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 502422528 bytes 2024-09-14T23:00:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 515169280 bytes 2024-09-14T23:00:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 568173568 bytes 2024-09-14T23:00:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 598000640 bytes 2024-09-14T23:00:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 610385920 bytes 2024-09-14T23:00:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 665313280 bytes 2024-09-14T23:00:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 717214720 bytes 2024-09-14T23:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 750638080 bytes 2024-09-14T23:00:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 758791168 bytes 2024-09-14T23:00:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 772557824 bytes 2024-09-14T23:00:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 809819136 bytes 2024-09-14T23:00:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 825157632 bytes 2024-09-14T23:00:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 884059136 bytes 2024-09-14T23:00:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 937203712 bytes 2024-09-14T23:00:59.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 963827712 bytes 2024-09-14T23:00:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 990722048 bytes 2024-09-14T23:00:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1029899264 bytes 2024-09-14T23:00:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1070386176 bytes 2024-09-14T23:00:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1135542272 bytes 2024-09-14T23:00:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1160898560 bytes 2024-09-14T23:00:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1186888704 bytes 2024-09-14T23:01:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1298617344 bytes 2024-09-14T23:01:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1423684608 bytes 2024-09-14T23:01:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1458196480 bytes 2024-09-14T23:01:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1528133632 bytes 2024-09-14T23:01:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1695202304 bytes 2024-09-14T23:01:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1744769024 bytes 2024-09-14T23:01:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1767824384 bytes 2024-09-14T23:01:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1823954944 bytes 2024-09-14T23:01:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1869393920 bytes 2024-09-14T23:01:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1878441984 bytes 2024-09-14T23:01:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1904675840 bytes 2024-09-14T23:01:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1970196480 bytes 2024-09-14T23:01:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2012672000 bytes 2024-09-14T23:01:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2078579712 bytes 2024-09-14T23:01:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2179003392 bytes 2024-09-14T23:01:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2221931520 bytes 2024-09-14T23:01:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2239707136 bytes 2024-09-14T23:01:02.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2299968512 bytes 2024-09-14T23:01:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2308271104 bytes 2024-09-14T23:01:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2367443968 bytes 2024-09-14T23:01:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2375379968 bytes 2024-09-14T23:01:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2438988800 bytes 2024-09-14T23:01:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2439612416 bytes 2024-09-14T23:01:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2505112576 bytes 2024-09-14T23:01:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2534139904 bytes 2024-09-14T23:01:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2549801984 bytes 2024-09-14T23:01:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2561764352 bytes 2024-09-14T23:01:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2619134976 bytes 2024-09-14T23:01:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2667485184 bytes 2024-09-14T23:01:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2706029568 bytes 2024-09-14T23:01:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2751008768 bytes 2024-09-14T23:01:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2824500224 bytes 2024-09-14T23:01:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2835311616 bytes 2024-09-14T23:01:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2845080576 bytes 2024-09-14T23:01:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2854043648 bytes 2024-09-14T23:01:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:01:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:01:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:01:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:01:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_3 2024-09-14T23:01:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-14T23:01:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:01:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:01:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:01:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:01:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2024-09-14T23:01:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:01:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:01:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-14T23:01:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:01:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-14T23:01:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:01:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-14T23:01:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_3': 2024-09-14T23:01:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:01:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:01:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:01:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1291c40db2a3 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1291c40db2a3 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:03 2024 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:03 2024 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:03 2024 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 2903cd12-fe5f-4679-a7de-9ede74f67058 2024-09-14T23:01:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:01:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:01:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-14T23:01:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:01:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:01:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=10 2024-09-14T23:01:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:01:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 22774784 bytes 2024-09-14T23:01:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 103347200 bytes 2024-09-14T23:01:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 135880704 bytes 2024-09-14T23:01:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 149620736 bytes 2024-09-14T23:01:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 150245376 bytes 2024-09-14T23:01:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 252704768 bytes 2024-09-14T23:01:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 336595968 bytes 2024-09-14T23:01:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 390693888 bytes 2024-09-14T23:01:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 401924096 bytes 2024-09-14T23:01:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 438815744 bytes 2024-09-14T23:01:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 509198336 bytes 2024-09-14T23:01:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 597364736 bytes 2024-09-14T23:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 645083136 bytes 2024-09-14T23:01:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 698717184 bytes 2024-09-14T23:01:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 717468672 bytes 2024-09-14T23:01:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 792079360 bytes 2024-09-14T23:01:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 827829248 bytes 2024-09-14T23:01:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 880535552 bytes 2024-09-14T23:01:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 953998336 bytes 2024-09-14T23:01:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 954606592 bytes 2024-09-14T23:01:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 972732416 bytes 2024-09-14T23:01:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1014632448 bytes 2024-09-14T23:01:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1143446528 bytes 2024-09-14T23:01:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1144040448 bytes 2024-09-14T23:01:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1205037056 bytes 2024-09-14T23:01:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1228153856 bytes 2024-09-14T23:01:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1243644928 bytes 2024-09-14T23:01:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1284497408 bytes 2024-09-14T23:01:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1317676032 bytes 2024-09-14T23:01:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1349603328 bytes 2024-09-14T23:01:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1399928832 bytes 2024-09-14T23:01:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1442755584 bytes 2024-09-14T23:01:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1458305024 bytes 2024-09-14T23:01:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1502282752 bytes 2024-09-14T23:01:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1523084288 bytes 2024-09-14T23:01:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1533393920 bytes 2024-09-14T23:01:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1560742912 bytes 2024-09-14T23:01:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1580026880 bytes 2024-09-14T23:01:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1589541888 bytes 2024-09-14T23:01:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1645942784 bytes 2024-09-14T23:01:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1673340928 bytes 2024-09-14T23:01:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1718103040 bytes 2024-09-14T23:01:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1742476288 bytes 2024-09-14T23:01:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1832521728 bytes 2024-09-14T23:01:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1855539200 bytes 2024-09-14T23:01:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1903065088 bytes 2024-09-14T23:01:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1944268800 bytes 2024-09-14T23:01:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2028058624 bytes 2024-09-14T23:01:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2056181760 bytes 2024-09-14T23:01:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2078329856 bytes 2024-09-14T23:01:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2100657152 bytes 2024-09-14T23:01:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2130287616 bytes 2024-09-14T23:01:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2161813504 bytes 2024-09-14T23:01:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2198990848 bytes 2024-09-14T23:01:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2265254912 bytes 2024-09-14T23:01:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2302384128 bytes 2024-09-14T23:01:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2326157312 bytes 2024-09-14T23:01:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2385605632 bytes 2024-09-14T23:01:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2469298176 bytes 2024-09-14T23:01:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2511809536 bytes 2024-09-14T23:01:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2599170048 bytes 2024-09-14T23:01:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2670183424 bytes 2024-09-14T23:01:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2731382784 bytes 2024-09-14T23:01:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2797632512 bytes 2024-09-14T23:01:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2815834112 bytes 2024-09-14T23:01:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2856551424 bytes 2024-09-14T23:01:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2938381312 bytes 2024-09-14T23:01:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2982901760 bytes 2024-09-14T23:01:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3038835712 bytes 2024-09-14T23:01:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3151629312 bytes 2024-09-14T23:01:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3171609600 bytes 2024-09-14T23:01:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3189695488 bytes 2024-09-14T23:01:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3214729216 bytes 2024-09-14T23:01:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3277979648 bytes 2024-09-14T23:01:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3293504512 bytes 2024-09-14T23:01:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3346663424 bytes 2024-09-14T23:01:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3382902784 bytes 2024-09-14T23:01:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3461914624 bytes 2024-09-14T23:01:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3495278592 bytes 2024-09-14T23:01:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3602895872 bytes 2024-09-14T23:01:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3675381760 bytes 2024-09-14T23:01:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3696729088 bytes 2024-09-14T23:01:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3728537600 bytes 2024-09-14T23:01:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3742387200 bytes 2024-09-14T23:01:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3772032000 bytes 2024-09-14T23:01:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_4 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:01:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2024-09-14T23:01:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:01:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:01:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-14T23:01:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:01:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-14T23:01:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:01:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_4': 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1299f62c6ea6 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1299f62c6ea6 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:14 2024 2024-09-14T23:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:14 2024 2024-09-14T23:01:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:14 2024 2024-09-14T23:01:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:01:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:01:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: d53f0a91-12f4-4ba8-846e-d09778cdee6d 2024-09-14T23:01:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:01:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:01:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-14T23:01:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:01:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:01:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T23:01:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:01:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:01:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 56614912 bytes 2024-09-14T23:01:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 82465792 bytes 2024-09-14T23:01:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 95804416 bytes 2024-09-14T23:01:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 130419712 bytes 2024-09-14T23:01:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 151083008 bytes 2024-09-14T23:01:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 178929664 bytes 2024-09-14T23:01:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 209806336 bytes 2024-09-14T23:01:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 288834560 bytes 2024-09-14T23:01:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 377596928 bytes 2024-09-14T23:01:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 412102656 bytes 2024-09-14T23:01:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 437050368 bytes 2024-09-14T23:01:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 570409984 bytes 2024-09-14T23:01:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 638126080 bytes 2024-09-14T23:01:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 638757888 bytes 2024-09-14T23:01:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 657199104 bytes 2024-09-14T23:01:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 688808960 bytes 2024-09-14T23:01:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 692720640 bytes 2024-09-14T23:01:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 693332992 bytes 2024-09-14T23:01:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 746382336 bytes 2024-09-14T23:01:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 800741376 bytes 2024-09-14T23:01:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 826187776 bytes 2024-09-14T23:01:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 896903168 bytes 2024-09-14T23:01:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 919997440 bytes 2024-09-14T23:01:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 965660672 bytes 2024-09-14T23:01:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1002936320 bytes 2024-09-14T23:01:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1026938880 bytes 2024-09-14T23:01:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1080223744 bytes 2024-09-14T23:01:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1080829952 bytes 2024-09-14T23:01:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1153588224 bytes 2024-09-14T23:01:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1179622400 bytes 2024-09-14T23:01:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1216728064 bytes 2024-09-14T23:01:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1217358848 bytes 2024-09-14T23:01:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1255738368 bytes 2024-09-14T23:01:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1279665152 bytes 2024-09-14T23:01:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1290991616 bytes 2024-09-14T23:01:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1358255104 bytes 2024-09-14T23:01:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1440964608 bytes 2024-09-14T23:01:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1463459840 bytes 2024-09-14T23:01:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1473423360 bytes 2024-09-14T23:01:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1546995712 bytes 2024-09-14T23:01:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1569188864 bytes 2024-09-14T23:01:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1611746304 bytes 2024-09-14T23:01:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1640805376 bytes 2024-09-14T23:01:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1679819776 bytes 2024-09-14T23:01:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1705089024 bytes 2024-09-14T23:01:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1728564224 bytes 2024-09-14T23:01:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1766918144 bytes 2024-09-14T23:01:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1812229120 bytes 2024-09-14T23:01:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1865066496 bytes 2024-09-14T23:01:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1928682496 bytes 2024-09-14T23:01:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 1948444672 bytes 2024-09-14T23:01:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2014765056 bytes 2024-09-14T23:01:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2057483264 bytes 2024-09-14T23:01:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2105105408 bytes 2024-09-14T23:01:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2145972224 bytes 2024-09-14T23:01:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:01:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:01:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_5 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-14T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-14T23:01:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:01:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-14T23:01:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_5': 2024-09-14T23:01:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:01:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:01:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:01:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12a167e97fee 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12a167e97fee 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:20 2024 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:20 2024 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:20 2024 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: cd71c6af-1235-405d-96be-3b0fea37da92 2024-09-14T23:01:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-14T23:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:01:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=9 2024-09-14T23:01:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:01:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:01:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 4713472 bytes 2024-09-14T23:01:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 56492032 bytes 2024-09-14T23:01:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 106048512 bytes 2024-09-14T23:01:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 132831232 bytes 2024-09-14T23:01:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 156665856 bytes 2024-09-14T23:01:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 180579328 bytes 2024-09-14T23:01:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 195264512 bytes 2024-09-14T23:01:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 256623616 bytes 2024-09-14T23:01:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 257236992 bytes 2024-09-14T23:01:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 284661760 bytes 2024-09-14T23:01:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 308234240 bytes 2024-09-14T23:01:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 352842752 bytes 2024-09-14T23:01:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 389052416 bytes 2024-09-14T23:01:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 471102464 bytes 2024-09-14T23:01:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 493698048 bytes 2024-09-14T23:01:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 527558656 bytes 2024-09-14T23:01:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 533133312 bytes 2024-09-14T23:01:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 613967872 bytes 2024-09-14T23:01:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 699530240 bytes 2024-09-14T23:01:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 754621440 bytes 2024-09-14T23:01:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 758914048 bytes 2024-09-14T23:01:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 770455552 bytes 2024-09-14T23:01:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 797413376 bytes 2024-09-14T23:01:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 841734144 bytes 2024-09-14T23:01:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 924073984 bytes 2024-09-14T23:01:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 996457472 bytes 2024-09-14T23:01:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1061552128 bytes 2024-09-14T23:01:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1119919104 bytes 2024-09-14T23:01:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1158087680 bytes 2024-09-14T23:01:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1223527424 bytes 2024-09-14T23:01:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1268592640 bytes 2024-09-14T23:01:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1322568704 bytes 2024-09-14T23:01:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1375077376 bytes 2024-09-14T23:01:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1375679488 bytes 2024-09-14T23:01:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1428398080 bytes 2024-09-14T23:01:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1500951552 bytes 2024-09-14T23:01:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1546566656 bytes 2024-09-14T23:01:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1571286016 bytes 2024-09-14T23:01:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1578641408 bytes 2024-09-14T23:01:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1641161728 bytes 2024-09-14T23:01:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1705917440 bytes 2024-09-14T23:01:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1757825024 bytes 2024-09-14T23:01:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1772991488 bytes 2024-09-14T23:01:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1845990400 bytes 2024-09-14T23:01:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1859798016 bytes 2024-09-14T23:01:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1950360576 bytes 2024-09-14T23:01:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1979251712 bytes 2024-09-14T23:01:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2010656768 bytes 2024-09-14T23:01:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2039977984 bytes 2024-09-14T23:01:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2102148096 bytes 2024-09-14T23:01:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2167564288 bytes 2024-09-14T23:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2180658176 bytes 2024-09-14T23:01:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2270303232 bytes 2024-09-14T23:01:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2270923776 bytes 2024-09-14T23:01:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2323842048 bytes 2024-09-14T23:01:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2324455424 bytes 2024-09-14T23:01:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2368386048 bytes 2024-09-14T23:01:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2434908160 bytes 2024-09-14T23:01:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2532042752 bytes 2024-09-14T23:01:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2571341824 bytes 2024-09-14T23:01:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2618598400 bytes 2024-09-14T23:01:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2626940928 bytes 2024-09-14T23:01:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2627874816 bytes 2024-09-14T23:01:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2689408000 bytes 2024-09-14T23:01:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2705201152 bytes 2024-09-14T23:01:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2787363840 bytes 2024-09-14T23:01:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2852624384 bytes 2024-09-14T23:01:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2874473472 bytes 2024-09-14T23:01:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2937616384 bytes 2024-09-14T23:01:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2956688384 bytes 2024-09-14T23:01:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3029705728 bytes 2024-09-14T23:01:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3035714560 bytes 2024-09-14T23:01:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3092207616 bytes 2024-09-14T23:01:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3132890112 bytes 2024-09-14T23:01:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3268783104 bytes 2024-09-14T23:01:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3297756160 bytes 2024-09-14T23:01:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3342004224 bytes 2024-09-14T23:01:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3358131200 bytes 2024-09-14T23:01:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3436046336 bytes 2024-09-14T23:01:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3476236288 bytes 2024-09-14T23:01:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3538632704 bytes 2024-09-14T23:01:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3604315136 bytes 2024-09-14T23:01:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3637787648 bytes 2024-09-14T23:01:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3693153280 bytes 2024-09-14T23:01:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3735059456 bytes 2024-09-14T23:01:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:01:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:01:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:01:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:01:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_6 2024-09-14T23:01:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-14T23:01:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:01:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:01:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:01:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:01:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2024-09-14T23:01:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:01:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:01:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-14T23:01:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:01:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-14T23:01:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:01:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-14T23:01:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_6': 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12aa48933417 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12aa48933417 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:30 2024 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:30 2024 2024-09-14T23:01:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:30 2024 2024-09-14T23:01:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:01:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:01:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 63095e72-2322-434d-9a80-ed8a5dcddbff 2024-09-14T23:01:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:01:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:01:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-14T23:01:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:01:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:01:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=11 2024-09-14T23:01:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:01:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:01:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 52644864 bytes 2024-09-14T23:01:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 78210048 bytes 2024-09-14T23:01:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 101907456 bytes 2024-09-14T23:01:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 145729536 bytes 2024-09-14T23:01:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 165724160 bytes 2024-09-14T23:01:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 185945088 bytes 2024-09-14T23:01:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 253498368 bytes 2024-09-14T23:01:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 280069120 bytes 2024-09-14T23:01:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 328458240 bytes 2024-09-14T23:01:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 343950336 bytes 2024-09-14T23:01:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 348106752 bytes 2024-09-14T23:01:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 367970304 bytes 2024-09-14T23:01:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 434194432 bytes 2024-09-14T23:01:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 482127872 bytes 2024-09-14T23:01:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 527125504 bytes 2024-09-14T23:01:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 562717696 bytes 2024-09-14T23:01:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 669999104 bytes 2024-09-14T23:01:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 675085312 bytes 2024-09-14T23:01:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 706754560 bytes 2024-09-14T23:01:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 749624320 bytes 2024-09-14T23:01:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 836368384 bytes 2024-09-14T23:01:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 868560896 bytes 2024-09-14T23:01:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 921480192 bytes 2024-09-14T23:01:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 944262144 bytes 2024-09-14T23:01:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 989715456 bytes 2024-09-14T23:01:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1050301440 bytes 2024-09-14T23:01:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1074174976 bytes 2024-09-14T23:01:34.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1133086720 bytes 2024-09-14T23:01:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1222026240 bytes 2024-09-14T23:01:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1229877248 bytes 2024-09-14T23:01:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1324771328 bytes 2024-09-14T23:01:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1395456000 bytes 2024-09-14T23:01:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1439354880 bytes 2024-09-14T23:01:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1453485056 bytes 2024-09-14T23:01:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1516574720 bytes 2024-09-14T23:01:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1555107840 bytes 2024-09-14T23:01:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1572135936 bytes 2024-09-14T23:01:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1634199552 bytes 2024-09-14T23:01:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1653551104 bytes 2024-09-14T23:01:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1692469248 bytes 2024-09-14T23:01:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1739950080 bytes 2024-09-14T23:01:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1863341056 bytes 2024-09-14T23:01:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1888789504 bytes 2024-09-14T23:01:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1960878080 bytes 2024-09-14T23:01:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2067678208 bytes 2024-09-14T23:01:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2094387200 bytes 2024-09-14T23:01:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2095010816 bytes 2024-09-14T23:01:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2168124416 bytes 2024-09-14T23:01:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2226010112 bytes 2024-09-14T23:01:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2226756608 bytes 2024-09-14T23:01:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2243047424 bytes 2024-09-14T23:01:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2253848576 bytes 2024-09-14T23:01:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2302170112 bytes 2024-09-14T23:01:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2312006656 bytes 2024-09-14T23:01:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2354619392 bytes 2024-09-14T23:01:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2407161856 bytes 2024-09-14T23:01:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2429071360 bytes 2024-09-14T23:01:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2460731392 bytes 2024-09-14T23:01:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2478552064 bytes 2024-09-14T23:01:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2517894144 bytes 2024-09-14T23:01:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2518515712 bytes 2024-09-14T23:01:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2546008064 bytes 2024-09-14T23:01:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2566949888 bytes 2024-09-14T23:01:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2579533824 bytes 2024-09-14T23:01:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2622459904 bytes 2024-09-14T23:01:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2646385664 bytes 2024-09-14T23:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2711438336 bytes 2024-09-14T23:01:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2771438592 bytes 2024-09-14T23:01:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2778734592 bytes 2024-09-14T23:01:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2788756480 bytes 2024-09-14T23:01:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2830201856 bytes 2024-09-14T23:01:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2881339392 bytes 2024-09-14T23:01:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2911896576 bytes 2024-09-14T23:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2912520192 bytes 2024-09-14T23:01:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 2951577600 bytes 2024-09-14T23:01:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 2979427328 bytes 2024-09-14T23:01:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3030162432 bytes 2024-09-14T23:01:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3126454272 bytes 2024-09-14T23:01:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3226992640 bytes 2024-09-14T23:01:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3281182720 bytes 2024-09-14T23:01:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3343468544 bytes 2024-09-14T23:01:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3449582592 bytes 2024-09-14T23:01:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3523320832 bytes 2024-09-14T23:01:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3552685056 bytes 2024-09-14T23:01:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3581858816 bytes 2024-09-14T23:01:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3595532288 bytes 2024-09-14T23:01:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3597084672 bytes 2024-09-14T23:01:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3625725952 bytes 2024-09-14T23:01:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3649087488 bytes 2024-09-14T23:01:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3690732544 bytes 2024-09-14T23:01:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3744397312 bytes 2024-09-14T23:01:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3747054592 bytes 2024-09-14T23:01:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3778152448 bytes 2024-09-14T23:01:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3849040896 bytes 2024-09-14T23:01:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 3917999104 bytes 2024-09-14T23:01:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 3991313408 bytes 2024-09-14T23:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4020046848 bytes 2024-09-14T23:01:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4049634304 bytes 2024-09-14T23:01:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4097309696 bytes 2024-09-14T23:01:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4106901504 bytes 2024-09-14T23:01:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4155813888 bytes 2024-09-14T23:01:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4194782208 bytes 2024-09-14T23:01:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4213808128 bytes 2024-09-14T23:01:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4250896384 bytes 2024-09-14T23:01:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4308313088 bytes 2024-09-14T23:01:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:01:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:01:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:01:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_7 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:01:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:01:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:01:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:01:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2024-09-14T23:01:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:01:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:01:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-14T23:01:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:01:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-14T23:01:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:01:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_7': 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12b3b868ddd8 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12b3b868ddd8 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:42 2024 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:42 2024 2024-09-14T23:01:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:42 2024 2024-09-14T23:01:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:01:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:01:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: b412e86d-c0d6-4c66-9336-1d3894abecba 2024-09-14T23:01:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:01:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:01:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-14T23:01:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:01:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:01:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=8 2024-09-14T23:01:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:01:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:01:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 35592192 bytes 2024-09-14T23:01:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 36182016 bytes 2024-09-14T23:01:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 123925504 bytes 2024-09-14T23:01:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 147257344 bytes 2024-09-14T23:01:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 193569792 bytes 2024-09-14T23:01:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 232514560 bytes 2024-09-14T23:01:43.898 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7037959635826636) > weight (0.25) 2024-09-14T23:01:43.898 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8245129498082127) > weight (0.25) 2024-09-14T23:01:43.899 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-14T23:01:43.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-14T23:01:43.899 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T23:01:43.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:01:43.897+0000 7fe8e41d6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 36906) UID: 1000 2024-09-14T23:01:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 236134400 bytes 2024-09-14T23:01:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 280170496 bytes 2024-09-14T23:01:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 313589760 bytes 2024-09-14T23:01:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 327291904 bytes 2024-09-14T23:01:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 345345024 bytes 2024-09-14T23:01:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 367846400 bytes 2024-09-14T23:01:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 436156416 bytes 2024-09-14T23:01:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 464665600 bytes 2024-09-14T23:01:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 511936512 bytes 2024-09-14T23:01:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 533576704 bytes 2024-09-14T23:01:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 617958400 bytes 2024-09-14T23:01:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 625862656 bytes 2024-09-14T23:01:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 636948480 bytes 2024-09-14T23:01:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 658412544 bytes 2024-09-14T23:01:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 687179776 bytes 2024-09-14T23:01:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 730726400 bytes 2024-09-14T23:01:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 789291008 bytes 2024-09-14T23:01:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 861070336 bytes 2024-09-14T23:01:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 914582528 bytes 2024-09-14T23:01:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 972523520 bytes 2024-09-14T23:01:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1010996224 bytes 2024-09-14T23:01:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1067544576 bytes 2024-09-14T23:01:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1127548928 bytes 2024-09-14T23:01:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1142163456 bytes 2024-09-14T23:01:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1144785920 bytes 2024-09-14T23:01:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1175102464 bytes 2024-09-14T23:01:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1205856256 bytes 2024-09-14T23:01:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1257589760 bytes 2024-09-14T23:01:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1290711040 bytes 2024-09-14T23:01:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1390737408 bytes 2024-09-14T23:01:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1408047104 bytes 2024-09-14T23:01:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1439978496 bytes 2024-09-14T23:01:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1460637696 bytes 2024-09-14T23:01:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1465962496 bytes 2024-09-14T23:01:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1512474624 bytes 2024-09-14T23:01:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1541246976 bytes 2024-09-14T23:01:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1607018496 bytes 2024-09-14T23:01:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1655393280 bytes 2024-09-14T23:01:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1724235776 bytes 2024-09-14T23:01:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1743021056 bytes 2024-09-14T23:01:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1757243392 bytes 2024-09-14T23:01:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1820897280 bytes 2024-09-14T23:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1871160320 bytes 2024-09-14T23:01:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1919048704 bytes 2024-09-14T23:01:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 1968654336 bytes 2024-09-14T23:01:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2024721408 bytes 2024-09-14T23:01:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2035237888 bytes 2024-09-14T23:01:48.900 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-14T23:01:48.900 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:01:48.900 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-14T23:01:48.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T23:01:48.901 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T23:01:48.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T23:01:48.904 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-14T23:01:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2046641152 bytes 2024-09-14T23:01:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2085572608 bytes 2024-09-14T23:01:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2162566144 bytes 2024-09-14T23:01:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2163165184 bytes 2024-09-14T23:01:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2229098496 bytes 2024-09-14T23:01:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2251144192 bytes 2024-09-14T23:01:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2318423040 bytes 2024-09-14T23:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2392855552 bytes 2024-09-14T23:01:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2441830400 bytes 2024-09-14T23:01:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2449876992 bytes 2024-09-14T23:01:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2479691776 bytes 2024-09-14T23:01:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2511892480 bytes 2024-09-14T23:01:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2575824896 bytes 2024-09-14T23:01:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2631351296 bytes 2024-09-14T23:01:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2676151296 bytes 2024-09-14T23:01:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2723019776 bytes 2024-09-14T23:01:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2723624960 bytes 2024-09-14T23:01:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2739968000 bytes 2024-09-14T23:01:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2745205760 bytes 2024-09-14T23:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2779550720 bytes 2024-09-14T23:01:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2801769472 bytes 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_8 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:01:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:01:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:01:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:01:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2024-09-14T23:01:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:01:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:01:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:01:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-14T23:01:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:01:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:01:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-14T23:01:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:01:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-14T23:01:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_8': 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12c47d075880 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12c47d075880 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:51 2024 2024-09-14T23:01:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:51 2024 2024-09-14T23:01:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:51 2024 2024-09-14T23:01:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:01:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:01:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 1183b733-6e8d-4fa0-bcc5-8cdba88bdd37 2024-09-14T23:01:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:01:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:01:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-14T23:01:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:01:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:01:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:01:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:01:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=10 2024-09-14T23:01:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:01:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:01:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 49297408 bytes 2024-09-14T23:01:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 49921024 bytes 2024-09-14T23:01:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 95252480 bytes 2024-09-14T23:01:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 102723584 bytes 2024-09-14T23:01:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 115283968 bytes 2024-09-14T23:01:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 140457984 bytes 2024-09-14T23:01:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 146521088 bytes 2024-09-14T23:01:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 220853248 bytes 2024-09-14T23:01:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 295928832 bytes 2024-09-14T23:01:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 331431936 bytes 2024-09-14T23:01:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 398971904 bytes 2024-09-14T23:01:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 417197056 bytes 2024-09-14T23:01:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 434271232 bytes 2024-09-14T23:01:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 477879296 bytes 2024-09-14T23:01:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 539564032 bytes 2024-09-14T23:01:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 562655232 bytes 2024-09-14T23:01:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 598948864 bytes 2024-09-14T23:01:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 599559168 bytes 2024-09-14T23:01:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 623801344 bytes 2024-09-14T23:01:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 714871808 bytes 2024-09-14T23:01:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 760318976 bytes 2024-09-14T23:01:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 768458752 bytes 2024-09-14T23:01:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 769192960 bytes 2024-09-14T23:01:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 838518784 bytes 2024-09-14T23:01:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 933377024 bytes 2024-09-14T23:01:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1024747520 bytes 2024-09-14T23:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1137281024 bytes 2024-09-14T23:01:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1168011264 bytes 2024-09-14T23:01:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1200411648 bytes 2024-09-14T23:01:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1201037312 bytes 2024-09-14T23:01:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1229327360 bytes 2024-09-14T23:01:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1290355712 bytes 2024-09-14T23:01:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1399596032 bytes 2024-09-14T23:01:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1405796352 bytes 2024-09-14T23:01:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1456352256 bytes 2024-09-14T23:01:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1518900224 bytes 2024-09-14T23:01:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1548906496 bytes 2024-09-14T23:01:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1586560000 bytes 2024-09-14T23:01:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1642079232 bytes 2024-09-14T23:01:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1659816960 bytes 2024-09-14T23:01:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1688267776 bytes 2024-09-14T23:01:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1744998400 bytes 2024-09-14T23:01:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1768189952 bytes 2024-09-14T23:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1800940544 bytes 2024-09-14T23:01:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1828300800 bytes 2024-09-14T23:01:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1880299520 bytes 2024-09-14T23:01:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1928841216 bytes 2024-09-14T23:01:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1975956480 bytes 2024-09-14T23:01:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2005231616 bytes 2024-09-14T23:01:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2043471872 bytes 2024-09-14T23:01:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2051318784 bytes 2024-09-14T23:01:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2078334976 bytes 2024-09-14T23:01:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2102039552 bytes 2024-09-14T23:01:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2119737344 bytes 2024-09-14T23:01:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2137931776 bytes 2024-09-14T23:01:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2150913024 bytes 2024-09-14T23:01:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2170936320 bytes 2024-09-14T23:01:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2213283840 bytes 2024-09-14T23:01:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2302204928 bytes 2024-09-14T23:01:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2339595264 bytes 2024-09-14T23:01:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2426161152 bytes 2024-09-14T23:01:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2486085632 bytes 2024-09-14T23:01:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2524188672 bytes 2024-09-14T23:01:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2548082688 bytes 2024-09-14T23:01:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2610198528 bytes 2024-09-14T23:01:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2671706112 bytes 2024-09-14T23:01:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2697190400 bytes 2024-09-14T23:01:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2746769408 bytes 2024-09-14T23:01:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2789457920 bytes 2024-09-14T23:01:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2843444224 bytes 2024-09-14T23:01:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2901637120 bytes 2024-09-14T23:02:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2924364800 bytes 2024-09-14T23:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2925806592 bytes 2024-09-14T23:02:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2971890688 bytes 2024-09-14T23:02:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 2981646336 bytes 2024-09-14T23:02:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3056769024 bytes 2024-09-14T23:02:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3114803200 bytes 2024-09-14T23:02:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3175051264 bytes 2024-09-14T23:02:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3203489792 bytes 2024-09-14T23:02:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3227295744 bytes 2024-09-14T23:02:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3263087616 bytes 2024-09-14T23:02:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3323659264 bytes 2024-09-14T23:02:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3338319872 bytes 2024-09-14T23:02:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3409580032 bytes 2024-09-14T23:02:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3438477312 bytes 2024-09-14T23:02:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3481801728 bytes 2024-09-14T23:02:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3546011648 bytes 2024-09-14T23:02:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3582466048 bytes 2024-09-14T23:02:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3676794880 bytes 2024-09-14T23:02:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3716760576 bytes 2024-09-14T23:02:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3781831680 bytes 2024-09-14T23:02:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3810293760 bytes 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_9 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:02:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:02:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:02:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2024-09-14T23:02:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:02:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:02:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-14T23:02:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:02:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-14T23:02:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:02:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-14T23:02:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_9': 2024-09-14T23:02:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:02:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12cd9b13a5c3 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12cd9b13a5c3 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:02:02 2024 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:02:02 2024 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:02:02 2024 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: e951d895-861e-422c-881e-e2fec0aaeafd 2024-09-14T23:02:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:02:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:02:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-14T23:02:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:02:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:02:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=10 2024-09-14T23:02:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:02:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:02:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 44956672 bytes 2024-09-14T23:02:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 64410624 bytes 2024-09-14T23:02:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 110524416 bytes 2024-09-14T23:02:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 153587712 bytes 2024-09-14T23:02:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 209224704 bytes 2024-09-14T23:02:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 250547200 bytes 2024-09-14T23:02:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 273415168 bytes 2024-09-14T23:02:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 319925248 bytes 2024-09-14T23:02:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 388959232 bytes 2024-09-14T23:02:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 454259712 bytes 2024-09-14T23:02:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 489214976 bytes 2024-09-14T23:02:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 519123968 bytes 2024-09-14T23:02:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 567820288 bytes 2024-09-14T23:02:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 579874816 bytes 2024-09-14T23:02:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 610385920 bytes 2024-09-14T23:02:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 632354816 bytes 2024-09-14T23:02:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 723290112 bytes 2024-09-14T23:02:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 758007808 bytes 2024-09-14T23:02:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 795065344 bytes 2024-09-14T23:02:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 817275904 bytes 2024-09-14T23:02:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 852700160 bytes 2024-09-14T23:02:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 897865728 bytes 2024-09-14T23:02:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 922028032 bytes 2024-09-14T23:02:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 948860928 bytes 2024-09-14T23:02:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 985864192 bytes 2024-09-14T23:02:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1007135744 bytes 2024-09-14T23:02:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1052312576 bytes 2024-09-14T23:02:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1102763008 bytes 2024-09-14T23:02:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1140772864 bytes 2024-09-14T23:02:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1149079552 bytes 2024-09-14T23:02:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1169583104 bytes 2024-09-14T23:02:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1212576768 bytes 2024-09-14T23:02:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1279631360 bytes 2024-09-14T23:02:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1334611968 bytes 2024-09-14T23:02:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1344348160 bytes 2024-09-14T23:02:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1382127616 bytes 2024-09-14T23:02:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1412818944 bytes 2024-09-14T23:02:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1516718080 bytes 2024-09-14T23:02:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1517335552 bytes 2024-09-14T23:02:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1529605120 bytes 2024-09-14T23:02:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1557446656 bytes 2024-09-14T23:02:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1595699200 bytes 2024-09-14T23:02:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1625053184 bytes 2024-09-14T23:02:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1696112640 bytes 2024-09-14T23:02:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1745567744 bytes 2024-09-14T23:02:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1777651712 bytes 2024-09-14T23:02:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1862188032 bytes 2024-09-14T23:02:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1883790336 bytes 2024-09-14T23:02:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1922557952 bytes 2024-09-14T23:02:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1958410240 bytes 2024-09-14T23:02:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2032056320 bytes 2024-09-14T23:02:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2041838592 bytes 2024-09-14T23:02:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2062846976 bytes 2024-09-14T23:02:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2086212608 bytes 2024-09-14T23:02:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2125644800 bytes 2024-09-14T23:02:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2126258176 bytes 2024-09-14T23:02:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2165551104 bytes 2024-09-14T23:02:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2221031424 bytes 2024-09-14T23:02:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2304034816 bytes 2024-09-14T23:02:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2333939712 bytes 2024-09-14T23:02:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2356430848 bytes 2024-09-14T23:02:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2415623168 bytes 2024-09-14T23:02:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2439424000 bytes 2024-09-14T23:02:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2474014720 bytes 2024-09-14T23:02:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2490292224 bytes 2024-09-14T23:02:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2518258688 bytes 2024-09-14T23:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2617206784 bytes 2024-09-14T23:02:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2617817088 bytes 2024-09-14T23:02:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2667246592 bytes 2024-09-14T23:02:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2712990720 bytes 2024-09-14T23:02:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2788372480 bytes 2024-09-14T23:02:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2822414336 bytes 2024-09-14T23:02:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2853445632 bytes 2024-09-14T23:02:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2903115776 bytes 2024-09-14T23:02:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 2915955712 bytes 2024-09-14T23:02:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 2950554624 bytes 2024-09-14T23:02:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 2977981440 bytes 2024-09-14T23:02:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3029490688 bytes 2024-09-14T23:02:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3037902848 bytes 2024-09-14T23:02:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3109173248 bytes 2024-09-14T23:02:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3198457856 bytes 2024-09-14T23:02:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3284975616 bytes 2024-09-14T23:02:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3344489472 bytes 2024-09-14T23:02:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3370349568 bytes 2024-09-14T23:02:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3382284288 bytes 2024-09-14T23:02:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3414926336 bytes 2024-09-14T23:02:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3446228992 bytes 2024-09-14T23:02:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3507739648 bytes 2024-09-14T23:02:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3548104704 bytes 2024-09-14T23:02:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3576007680 bytes 2024-09-14T23:02:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3596430336 bytes 2024-09-14T23:02:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:02:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:02:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:02:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_10 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-14T23:02:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2024-09-14T23:02:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:02:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:02:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-14T23:02:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:02:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-14T23:02:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:02:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-14T23:02:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_10': 2024-09-14T23:02:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:02:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:02:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:02:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12d82fb21dad 2024-09-14T23:02:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12d82fb21dad 2024-09-14T23:02:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:02:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:02:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:02:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:02:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:02:13 2024 2024-09-14T23:02:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:02:13 2024 2024-09-14T23:02:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:02:13 2024 2024-09-14T23:02:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:02:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:02:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: fd912f05-09a2-4d03-a711-51d6bfba6b89 2024-09-14T23:02:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:02:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:02:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-14T23:02:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:02:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:02:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=9 2024-09-14T23:02:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:02:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:02:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 22027264 bytes 2024-09-14T23:02:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 43621376 bytes 2024-09-14T23:02:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 51255296 bytes 2024-09-14T23:02:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 117327872 bytes 2024-09-14T23:02:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 153947136 bytes 2024-09-14T23:02:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 177575936 bytes 2024-09-14T23:02:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 244944896 bytes 2024-09-14T23:02:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 326922240 bytes 2024-09-14T23:02:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 354812928 bytes 2024-09-14T23:02:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 390689792 bytes 2024-09-14T23:02:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 405939200 bytes 2024-09-14T23:02:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 485431296 bytes 2024-09-14T23:02:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 525803520 bytes 2024-09-14T23:02:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 572095488 bytes 2024-09-14T23:02:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 668993536 bytes 2024-09-14T23:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 748220416 bytes 2024-09-14T23:02:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 782319616 bytes 2024-09-14T23:02:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 819892224 bytes 2024-09-14T23:02:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 872996864 bytes 2024-09-14T23:02:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 907176960 bytes 2024-09-14T23:02:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 945356800 bytes 2024-09-14T23:02:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 969268224 bytes 2024-09-14T23:02:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1035405312 bytes 2024-09-14T23:02:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1143098368 bytes 2024-09-14T23:02:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1160348672 bytes 2024-09-14T23:02:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1185080320 bytes 2024-09-14T23:02:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1222360064 bytes 2024-09-14T23:02:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1240317952 bytes 2024-09-14T23:02:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1286587392 bytes 2024-09-14T23:02:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1301356544 bytes 2024-09-14T23:02:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1380586496 bytes 2024-09-14T23:02:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1392831488 bytes 2024-09-14T23:02:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1438482432 bytes 2024-09-14T23:02:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1463222272 bytes 2024-09-14T23:02:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1468666880 bytes 2024-09-14T23:02:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1500381184 bytes 2024-09-14T23:02:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1524201472 bytes 2024-09-14T23:02:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1575071744 bytes 2024-09-14T23:02:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1627254784 bytes 2024-09-14T23:02:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1646230528 bytes 2024-09-14T23:02:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1704602624 bytes 2024-09-14T23:02:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1767020544 bytes 2024-09-14T23:02:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1868695552 bytes 2024-09-14T23:02:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1896981504 bytes 2024-09-14T23:02:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1953863680 bytes 2024-09-14T23:02:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1969562624 bytes 2024-09-14T23:02:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1989506048 bytes 2024-09-14T23:02:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2058053632 bytes 2024-09-14T23:02:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2103425024 bytes 2024-09-14T23:02:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2139930624 bytes 2024-09-14T23:02:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2155921408 bytes 2024-09-14T23:02:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2198814720 bytes 2024-09-14T23:02:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2223080448 bytes 2024-09-14T23:02:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2250961920 bytes 2024-09-14T23:02:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2251566080 bytes 2024-09-14T23:02:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2285582336 bytes 2024-09-14T23:02:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2398034944 bytes 2024-09-14T23:02:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2407435264 bytes 2024-09-14T23:02:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2491126784 bytes 2024-09-14T23:02:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2539951104 bytes 2024-09-14T23:02:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2576533504 bytes 2024-09-14T23:02:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2642480128 bytes 2024-09-14T23:02:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2706554880 bytes 2024-09-14T23:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2721959936 bytes 2024-09-14T23:02:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2753941504 bytes 2024-09-14T23:02:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2777193472 bytes 2024-09-14T23:02:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2804200448 bytes 2024-09-14T23:02:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2841046016 bytes 2024-09-14T23:02:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2899584000 bytes 2024-09-14T23:02:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2965428224 bytes 2024-09-14T23:02:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2993967104 bytes 2024-09-14T23:02:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3001768960 bytes 2024-09-14T23:02:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3063035904 bytes 2024-09-14T23:02:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3108226048 bytes 2024-09-14T23:02:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3111020544 bytes 2024-09-14T23:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3154042880 bytes 2024-09-14T23:02:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3200300032 bytes 2024-09-14T23:02:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3200923648 bytes 2024-09-14T23:02:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3224531968 bytes 2024-09-14T23:02:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3277624320 bytes 2024-09-14T23:02:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:02:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:02:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:02:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:02:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_11 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-14T23:02:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:02:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-14T23:02:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_11': 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12e0ef7ec2fb 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12e0ef7ec2fb 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:02:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:02:23 2024 2024-09-14T23:02:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:02:23 2024 2024-09-14T23:02:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:02:23 2024 2024-09-14T23:02:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:02:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:02:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 3d92657a-d871-4f0e-87e5-49062e513296 2024-09-14T23:02:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:02:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:02:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-14T23:02:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:02:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:02:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=13 2024-09-14T23:02:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:02:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:02:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 29027328 bytes 2024-09-14T23:02:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 44677120 bytes 2024-09-14T23:02:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 108725248 bytes 2024-09-14T23:02:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 109334528 bytes 2024-09-14T23:02:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 178475008 bytes 2024-09-14T23:02:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 208786432 bytes 2024-09-14T23:02:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 234789888 bytes 2024-09-14T23:02:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 273335296 bytes 2024-09-14T23:02:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 307763200 bytes 2024-09-14T23:02:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 308366336 bytes 2024-09-14T23:02:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 333801472 bytes 2024-09-14T23:02:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 362178560 bytes 2024-09-14T23:02:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 375252992 bytes 2024-09-14T23:02:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 470316032 bytes 2024-09-14T23:02:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 506729472 bytes 2024-09-14T23:02:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 516882432 bytes 2024-09-14T23:02:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 576277504 bytes 2024-09-14T23:02:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 594865152 bytes 2024-09-14T23:02:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 675717120 bytes 2024-09-14T23:02:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 703124480 bytes 2024-09-14T23:02:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 747232256 bytes 2024-09-14T23:02:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 794804224 bytes 2024-09-14T23:02:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 812108800 bytes 2024-09-14T23:02:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 890478592 bytes 2024-09-14T23:02:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 977264640 bytes 2024-09-14T23:02:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1034670080 bytes 2024-09-14T23:02:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1086771200 bytes 2024-09-14T23:02:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1140360192 bytes 2024-09-14T23:02:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1183131648 bytes 2024-09-14T23:02:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1183757312 bytes 2024-09-14T23:02:27.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1223649280 bytes 2024-09-14T23:02:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1277816832 bytes 2024-09-14T23:02:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1297752064 bytes 2024-09-14T23:02:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1361822720 bytes 2024-09-14T23:02:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1419729920 bytes 2024-09-14T23:02:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1449971712 bytes 2024-09-14T23:02:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1556862976 bytes 2024-09-14T23:02:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1611934720 bytes 2024-09-14T23:02:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1636976640 bytes 2024-09-14T23:02:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1685123072 bytes 2024-09-14T23:02:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1724917760 bytes 2024-09-14T23:02:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1765564416 bytes 2024-09-14T23:02:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1801759744 bytes 2024-09-14T23:02:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1835102208 bytes 2024-09-14T23:02:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1860571136 bytes 2024-09-14T23:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1884361728 bytes 2024-09-14T23:02:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1960928256 bytes 2024-09-14T23:02:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1998260224 bytes 2024-09-14T23:02:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2023163904 bytes 2024-09-14T23:02:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2067192832 bytes 2024-09-14T23:02:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2081708032 bytes 2024-09-14T23:02:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2124378112 bytes 2024-09-14T23:02:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2169706496 bytes 2024-09-14T23:02:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2184761344 bytes 2024-09-14T23:02:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2229779456 bytes 2024-09-14T23:02:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2316149760 bytes 2024-09-14T23:02:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2329823232 bytes 2024-09-14T23:02:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2363745280 bytes 2024-09-14T23:02:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2406155264 bytes 2024-09-14T23:02:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2486681600 bytes 2024-09-14T23:02:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2531344384 bytes 2024-09-14T23:02:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2588424192 bytes 2024-09-14T23:02:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2626483200 bytes 2024-09-14T23:02:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2638896128 bytes 2024-09-14T23:02:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2639536128 bytes 2024-09-14T23:02:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2669628416 bytes 2024-09-14T23:02:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2695705600 bytes 2024-09-14T23:02:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2806407168 bytes 2024-09-14T23:02:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2833993728 bytes 2024-09-14T23:02:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2847482880 bytes 2024-09-14T23:02:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2889312256 bytes 2024-09-14T23:02:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2968111104 bytes 2024-09-14T23:02:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3017517056 bytes 2024-09-14T23:02:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3033357312 bytes 2024-09-14T23:02:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3050281984 bytes 2024-09-14T23:02:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3083830272 bytes 2024-09-14T23:02:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3100324864 bytes 2024-09-14T23:02:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3132060672 bytes 2024-09-14T23:02:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3170355200 bytes 2024-09-14T23:02:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3208144896 bytes 2024-09-14T23:02:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3276154880 bytes 2024-09-14T23:02:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3311833088 bytes 2024-09-14T23:02:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3438809088 bytes 2024-09-14T23:02:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3453255680 bytes 2024-09-14T23:02:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3520395264 bytes 2024-09-14T23:02:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3588579328 bytes 2024-09-14T23:02:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3638077440 bytes 2024-09-14T23:02:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3729325056 bytes 2024-09-14T23:02:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3745995776 bytes 2024-09-14T23:02:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3765100544 bytes 2024-09-14T23:02:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3805081600 bytes 2024-09-14T23:02:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3805691904 bytes 2024-09-14T23:02:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3866717184 bytes 2024-09-14T23:02:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3916319744 bytes 2024-09-14T23:02:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4021302272 bytes 2024-09-14T23:02:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4111198208 bytes 2024-09-14T23:02:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4141349888 bytes 2024-09-14T23:02:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4181465088 bytes 2024-09-14T23:02:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4226118656 bytes 2024-09-14T23:02:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4271897600 bytes 2024-09-14T23:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4312920064 bytes 2024-09-14T23:02:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4366706688 bytes 2024-09-14T23:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4414740480 bytes 2024-09-14T23:02:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4463081472 bytes 2024-09-14T23:02:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4499089408 bytes 2024-09-14T23:02:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4567062528 bytes 2024-09-14T23:02:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4568432640 bytes 2024-09-14T23:02:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4657378304 bytes 2024-09-14T23:02:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 4712739840 bytes 2024-09-14T23:02:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11000 IOs, 4750824448 bytes 2024-09-14T23:02:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11100 IOs, 4776536064 bytes 2024-09-14T23:02:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11200 IOs, 4835043328 bytes 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_12 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:02:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:02:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:02:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2024-09-14T23:02:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:02:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:02:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-14T23:02:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:02:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-14T23:02:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:02:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_12': 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12e97a49cdee 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12e97a49cdee 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:02:37 2024 2024-09-14T23:02:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:02:37 2024 2024-09-14T23:02:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:02:37 2024 2024-09-14T23:02:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:02:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:02:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 64ee2f3d-5dac-4606-aaf0-b005a4416265 2024-09-14T23:02:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:02:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:02:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-14T23:02:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:02:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:02:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=12 2024-09-14T23:02:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:02:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:02:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 25159680 bytes 2024-09-14T23:02:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 25772032 bytes 2024-09-14T23:02:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 116834304 bytes 2024-09-14T23:02:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 229373952 bytes 2024-09-14T23:02:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 250031104 bytes 2024-09-14T23:02:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 275407872 bytes 2024-09-14T23:02:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 352828416 bytes 2024-09-14T23:02:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 383595520 bytes 2024-09-14T23:02:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 439268352 bytes 2024-09-14T23:02:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 465011712 bytes 2024-09-14T23:02:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 528640000 bytes 2024-09-14T23:02:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 576396288 bytes 2024-09-14T23:02:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 603705344 bytes 2024-09-14T23:02:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 620271616 bytes 2024-09-14T23:02:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 666928128 bytes 2024-09-14T23:02:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 689911808 bytes 2024-09-14T23:02:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 758940672 bytes 2024-09-14T23:02:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 790133760 bytes 2024-09-14T23:02:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 833863680 bytes 2024-09-14T23:02:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 892165120 bytes 2024-09-14T23:02:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 967370752 bytes 2024-09-14T23:02:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1026695168 bytes 2024-09-14T23:02:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1094165504 bytes 2024-09-14T23:02:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1147376640 bytes 2024-09-14T23:02:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1208176640 bytes 2024-09-14T23:02:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1264371712 bytes 2024-09-14T23:02:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1344119808 bytes 2024-09-14T23:02:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1380961280 bytes 2024-09-14T23:02:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1408772096 bytes 2024-09-14T23:02:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1497423872 bytes 2024-09-14T23:02:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1562805248 bytes 2024-09-14T23:02:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1625409536 bytes 2024-09-14T23:02:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1648104448 bytes 2024-09-14T23:02:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1660098560 bytes 2024-09-14T23:02:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1716438016 bytes 2024-09-14T23:02:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1740153856 bytes 2024-09-14T23:02:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1804570624 bytes 2024-09-14T23:02:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1871194112 bytes 2024-09-14T23:02:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1871795200 bytes 2024-09-14T23:02:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1927404544 bytes 2024-09-14T23:02:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1959216128 bytes 2024-09-14T23:02:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1970740224 bytes 2024-09-14T23:02:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 2009038848 bytes 2024-09-14T23:02:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 2065501184 bytes 2024-09-14T23:02:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2124183552 bytes 2024-09-14T23:02:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2165182464 bytes 2024-09-14T23:02:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2178637824 bytes 2024-09-14T23:02:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2204825600 bytes 2024-09-14T23:02:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2281829376 bytes 2024-09-14T23:02:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2326979584 bytes 2024-09-14T23:02:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2362322944 bytes 2024-09-14T23:02:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2385595392 bytes 2024-09-14T23:02:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2396932096 bytes 2024-09-14T23:02:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2462409728 bytes 2024-09-14T23:02:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2491074560 bytes 2024-09-14T23:02:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2508258304 bytes 2024-09-14T23:02:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2561339392 bytes 2024-09-14T23:02:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2639895552 bytes 2024-09-14T23:02:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2679364608 bytes 2024-09-14T23:02:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2700865536 bytes 2024-09-14T23:02:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2710815744 bytes 2024-09-14T23:02:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2751240192 bytes 2024-09-14T23:02:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2786667520 bytes 2024-09-14T23:02:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2801238016 bytes 2024-09-14T23:02:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2899178496 bytes 2024-09-14T23:02:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2899825664 bytes 2024-09-14T23:02:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2942687232 bytes 2024-09-14T23:02:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 3008793600 bytes 2024-09-14T23:02:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3052883968 bytes 2024-09-14T23:02:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3090011136 bytes 2024-09-14T23:02:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3125404672 bytes 2024-09-14T23:02:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3129269248 bytes 2024-09-14T23:02:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3179709440 bytes 2024-09-14T23:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3191378944 bytes 2024-09-14T23:02:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3245583360 bytes 2024-09-14T23:02:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3306762240 bytes 2024-09-14T23:02:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3376630784 bytes 2024-09-14T23:02:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3386089472 bytes 2024-09-14T23:02:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3428171776 bytes 2024-09-14T23:02:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3469723648 bytes 2024-09-14T23:02:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3518303232 bytes 2024-09-14T23:02:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3568137216 bytes 2024-09-14T23:02:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3592159232 bytes 2024-09-14T23:02:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3665413120 bytes 2024-09-14T23:02:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3700179968 bytes 2024-09-14T23:02:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3818500096 bytes 2024-09-14T23:02:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3880230912 bytes 2024-09-14T23:02:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3926581248 bytes 2024-09-14T23:02:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3982852096 bytes 2024-09-14T23:02:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 4011930624 bytes 2024-09-14T23:02:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 4036288512 bytes 2024-09-14T23:02:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 4076995584 bytes 2024-09-14T23:02:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 4094202880 bytes 2024-09-14T23:02:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4159894528 bytes 2024-09-14T23:02:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4228745216 bytes 2024-09-14T23:02:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4262980608 bytes 2024-09-14T23:02:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4300461056 bytes 2024-09-14T23:02:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4302153728 bytes 2024-09-14T23:02:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4343746560 bytes 2024-09-14T23:02:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4395022336 bytes 2024-09-14T23:02:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4440918016 bytes 2024-09-14T23:02:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4483334144 bytes 2024-09-14T23:02:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4495380480 bytes 2024-09-14T23:02:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4520725504 bytes 2024-09-14T23:02:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_13 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:02:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:02:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2024-09-14T23:02:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-14T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-14T23:02:50.906 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.30538311065317547) > weight (0.25) 2024-09-14T23:02:50.906 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-14T23:02:50.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-14T23:02:50.907 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T23:02:50.907 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-14T23:02:50.907 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:02:50.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:02:50.906+0000 7fb330f77640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 36798) UID: 1000 2024-09-14T23:02:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:02:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-14T23:02:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_13': 2024-09-14T23:02:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:02:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:02:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:02:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12f12cbfcc93 2024-09-14T23:02:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12f12cbfcc93 2024-09-14T23:02:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:02:50 2024 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:02:50 2024 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:02:50 2024 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: ca7825af-19b6-473a-a8e5-328e157a8d1a 2024-09-14T23:02:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:02:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:02:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-14T23:02:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:02:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:02:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:02:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:02:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=10 2024-09-14T23:02:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:02:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:02:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 25677824 bytes 2024-09-14T23:02:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 140565504 bytes 2024-09-14T23:02:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 141174784 bytes 2024-09-14T23:02:52.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:02:52.000+0000 7f16795f7640 -1 rbd::mirror::ImageReplayer: 0x55e358480280 [3/13ef0e31-a893-45b8-a2c7-d21116cdc208] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:02:52.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:02:52.000+0000 7f1679df8640 -1 rbd::mirror::ImageReplayer: 0x55e358480280 [3/13ef0e31-a893-45b8-a2c7-d21116cdc208] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:02:52.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:02:52.000+0000 7f16795f7640 -1 rbd::mirror::ImageReplayer: 0x55e358da6a00 [3/fd912f05-09a2-4d03-a711-51d6bfba6b89] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:02:52.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:02:52.000+0000 7f1679df8640 -1 rbd::mirror::ImageReplayer: 0x55e358da6a00 [3/fd912f05-09a2-4d03-a711-51d6bfba6b89] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:02:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 196166656 bytes 2024-09-14T23:02:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 289712128 bytes 2024-09-14T23:02:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 309954560 bytes 2024-09-14T23:02:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 347381760 bytes 2024-09-14T23:02:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 374096896 bytes 2024-09-14T23:02:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 406882304 bytes 2024-09-14T23:02:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 446390272 bytes 2024-09-14T23:02:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 525521920 bytes 2024-09-14T23:02:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 567748608 bytes 2024-09-14T23:02:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 591098880 bytes 2024-09-14T23:02:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 619451392 bytes 2024-09-14T23:02:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 676711424 bytes 2024-09-14T23:02:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 702836736 bytes 2024-09-14T23:02:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 738806784 bytes 2024-09-14T23:02:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 762614784 bytes 2024-09-14T23:02:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 823013376 bytes 2024-09-14T23:02:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 865271808 bytes 2024-09-14T23:02:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 889076736 bytes 2024-09-14T23:02:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 956158976 bytes 2024-09-14T23:02:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1001684992 bytes 2024-09-14T23:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1029746688 bytes 2024-09-14T23:02:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1078545408 bytes 2024-09-14T23:02:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1081842688 bytes 2024-09-14T23:02:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1087754240 bytes 2024-09-14T23:02:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1108624384 bytes 2024-09-14T23:02:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1139077120 bytes 2024-09-14T23:02:54.811 INFO:tasks.ceph.cluster2.osd.0.smithi089.stderr:2024-09-14T23:02:54.810+0000 7fc768a4a640 -1 reset not still connected to 0x56493882c1a0 2024-09-14T23:02:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1172973568 bytes 2024-09-14T23:02:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1197970432 bytes 2024-09-14T23:02:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1216984064 bytes 2024-09-14T23:02:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1281537024 bytes 2024-09-14T23:02:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1308865536 bytes 2024-09-14T23:02:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1339212800 bytes 2024-09-14T23:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1426596864 bytes 2024-09-14T23:02:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1517237248 bytes 2024-09-14T23:02:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1541989376 bytes 2024-09-14T23:02:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1560999936 bytes 2024-09-14T23:02:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1599523840 bytes 2024-09-14T23:02:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1628738560 bytes 2024-09-14T23:02:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1685226496 bytes 2024-09-14T23:02:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1709285376 bytes 2024-09-14T23:02:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1842625536 bytes 2024-09-14T23:02:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1898352640 bytes 2024-09-14T23:02:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1919637504 bytes 2024-09-14T23:02:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1952515072 bytes 2024-09-14T23:02:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1974560768 bytes 2024-09-14T23:02:57.010 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-14T23:02:57.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T23:02:57.010 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T23:02:57.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T23:02:57.013 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-09-14T23:02:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2032688128 bytes 2024-09-14T23:02:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2120053760 bytes 2024-09-14T23:02:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2144801792 bytes 2024-09-14T23:02:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2160715776 bytes 2024-09-14T23:02:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2245977088 bytes 2024-09-14T23:02:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2278526976 bytes 2024-09-14T23:02:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2313961472 bytes 2024-09-14T23:02:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2405282816 bytes 2024-09-14T23:02:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2453982208 bytes 2024-09-14T23:02:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2480230400 bytes 2024-09-14T23:02:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2504593408 bytes 2024-09-14T23:02:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2605432832 bytes 2024-09-14T23:02:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2626944000 bytes 2024-09-14T23:02:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2630145024 bytes 2024-09-14T23:02:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2665422848 bytes 2024-09-14T23:02:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2776838144 bytes 2024-09-14T23:02:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2851603456 bytes 2024-09-14T23:02:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2915235840 bytes 2024-09-14T23:02:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2942493696 bytes 2024-09-14T23:02:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 3041047552 bytes 2024-09-14T23:02:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3104317440 bytes 2024-09-14T23:02:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3173150720 bytes 2024-09-14T23:03:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3244533760 bytes 2024-09-14T23:03:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3373729792 bytes 2024-09-14T23:03:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3443779584 bytes 2024-09-14T23:03:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3517644800 bytes 2024-09-14T23:03:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3597383680 bytes 2024-09-14T23:03:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3642468352 bytes 2024-09-14T23:03:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3664266240 bytes 2024-09-14T23:03:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3709949952 bytes 2024-09-14T23:03:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3728935936 bytes 2024-09-14T23:03:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3785525248 bytes 2024-09-14T23:03:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3803172864 bytes 2024-09-14T23:03:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:03:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:03:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:03:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_14 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-14T23:03:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2024-09-14T23:03:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:03:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:03:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-14T23:03:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:03:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-14T23:03:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:03:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-14T23:03:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_14': 2024-09-14T23:03:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:03:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:03:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:03:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13002ded4da7 2024-09-14T23:03:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13002ded4da7 2024-09-14T23:03:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:01 2024 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:01 2024 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:01 2024 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 45c698d2-2f4b-4209-84b4-d26cd2e23680 2024-09-14T23:03:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:03:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:03:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-14T23:03:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:03:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:03:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T23:03:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:03:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:03:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 152421376 bytes 2024-09-14T23:03:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 198601728 bytes 2024-09-14T23:03:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 236163072 bytes 2024-09-14T23:03:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 240656384 bytes 2024-09-14T23:03:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 325263360 bytes 2024-09-14T23:03:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 382851072 bytes 2024-09-14T23:03:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 401507328 bytes 2024-09-14T23:03:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 427227136 bytes 2024-09-14T23:03:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 505710592 bytes 2024-09-14T23:03:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 558907392 bytes 2024-09-14T23:03:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 559537152 bytes 2024-09-14T23:03:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 580928512 bytes 2024-09-14T23:03:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 590431232 bytes 2024-09-14T23:03:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 643930112 bytes 2024-09-14T23:03:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 722507776 bytes 2024-09-14T23:03:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 753265664 bytes 2024-09-14T23:03:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 873356288 bytes 2024-09-14T23:03:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 896966656 bytes 2024-09-14T23:03:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 962832384 bytes 2024-09-14T23:03:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 1006793728 bytes 2024-09-14T23:03:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 1051232256 bytes 2024-09-14T23:03:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1100531712 bytes 2024-09-14T23:03:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1199067136 bytes 2024-09-14T23:03:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1306535936 bytes 2024-09-14T23:03:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1345231872 bytes 2024-09-14T23:03:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1387312128 bytes 2024-09-14T23:03:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1408425984 bytes 2024-09-14T23:03:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1427866624 bytes 2024-09-14T23:03:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1501232128 bytes 2024-09-14T23:03:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1515355136 bytes 2024-09-14T23:03:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1591575552 bytes 2024-09-14T23:03:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1616245760 bytes 2024-09-14T23:03:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1667259392 bytes 2024-09-14T23:03:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1690050560 bytes 2024-09-14T23:03:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1749859328 bytes 2024-09-14T23:03:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1768595456 bytes 2024-09-14T23:03:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1800284160 bytes 2024-09-14T23:03:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1873043456 bytes 2024-09-14T23:03:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1946155008 bytes 2024-09-14T23:03:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1996525568 bytes 2024-09-14T23:03:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 2027202560 bytes 2024-09-14T23:03:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 2078752768 bytes 2024-09-14T23:03:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 2110701568 bytes 2024-09-14T23:03:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 2137216000 bytes 2024-09-14T23:03:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2173901824 bytes 2024-09-14T23:03:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2227738624 bytes 2024-09-14T23:03:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2252054528 bytes 2024-09-14T23:03:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2300796928 bytes 2024-09-14T23:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2316275712 bytes 2024-09-14T23:03:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2354333696 bytes 2024-09-14T23:03:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2395738112 bytes 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_15 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:03:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-14T23:03:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:03:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-14T23:03:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_15': 2024-09-14T23:03:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:03:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:03:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:03:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13081789cac1 2024-09-14T23:03:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13081789cac1 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:08 2024 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:08 2024 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:08 2024 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 77c7ed00-9d21-46b0-b7ed-332bbd914227 2024-09-14T23:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:03:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:03:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-14T23:03:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:03:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:03:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=8 2024-09-14T23:03:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:03:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:03:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 47165440 bytes 2024-09-14T23:03:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 117925888 bytes 2024-09-14T23:03:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 123851776 bytes 2024-09-14T23:03:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 180171776 bytes 2024-09-14T23:03:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 227040256 bytes 2024-09-14T23:03:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 227656704 bytes 2024-09-14T23:03:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 248049664 bytes 2024-09-14T23:03:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 251062272 bytes 2024-09-14T23:03:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 343104512 bytes 2024-09-14T23:03:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 387398656 bytes 2024-09-14T23:03:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 417188864 bytes 2024-09-14T23:03:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 460192768 bytes 2024-09-14T23:03:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 501595136 bytes 2024-09-14T23:03:10.904 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2768082033524404) > weight (0.25) 2024-09-14T23:03:10.904 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5640195146131529) > weight (0.25) 2024-09-14T23:03:10.904 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6573784423899276) > weight (0.25) 2024-09-14T23:03:10.904 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6659846398225975) > weight (0.25) 2024-09-14T23:03:10.904 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-14T23:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 555650048 bytes 2024-09-14T23:03:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 560181248 bytes 2024-09-14T23:03:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 630234112 bytes 2024-09-14T23:03:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 663204864 bytes 2024-09-14T23:03:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 707422208 bytes 2024-09-14T23:03:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 799794176 bytes 2024-09-14T23:03:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 826153984 bytes 2024-09-14T23:03:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 882227200 bytes 2024-09-14T23:03:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 896593920 bytes 2024-09-14T23:03:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 922064896 bytes 2024-09-14T23:03:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 934855680 bytes 2024-09-14T23:03:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1035867136 bytes 2024-09-14T23:03:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1037869056 bytes 2024-09-14T23:03:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1112753152 bytes 2024-09-14T23:03:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1169881088 bytes 2024-09-14T23:03:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1213410304 bytes 2024-09-14T23:03:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1241244672 bytes 2024-09-14T23:03:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1271634944 bytes 2024-09-14T23:03:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1306217472 bytes 2024-09-14T23:03:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1318585344 bytes 2024-09-14T23:03:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1377278976 bytes 2024-09-14T23:03:13.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1400774656 bytes 2024-09-14T23:03:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1407255552 bytes 2024-09-14T23:03:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1417017344 bytes 2024-09-14T23:03:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1487878144 bytes 2024-09-14T23:03:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1512047616 bytes 2024-09-14T23:03:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1584614400 bytes 2024-09-14T23:03:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1618064384 bytes 2024-09-14T23:03:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1674980352 bytes 2024-09-14T23:03:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1727279104 bytes 2024-09-14T23:03:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1765464064 bytes 2024-09-14T23:03:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1797735424 bytes 2024-09-14T23:03:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1889263616 bytes 2024-09-14T23:03:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1922665472 bytes 2024-09-14T23:03:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1983864832 bytes 2024-09-14T23:03:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2046883840 bytes 2024-09-14T23:03:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2108214272 bytes 2024-09-14T23:03:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2166521856 bytes 2024-09-14T23:03:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2250254336 bytes 2024-09-14T23:03:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2257075200 bytes 2024-09-14T23:03:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2290574336 bytes 2024-09-14T23:03:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2345398272 bytes 2024-09-14T23:03:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2360376320 bytes 2024-09-14T23:03:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2377863168 bytes 2024-09-14T23:03:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2424943616 bytes 2024-09-14T23:03:16.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2467844096 bytes 2024-09-14T23:03:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2468472832 bytes 2024-09-14T23:03:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2506998784 bytes 2024-09-14T23:03:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2530437120 bytes 2024-09-14T23:03:16.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2567863296 bytes 2024-09-14T23:03:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2605817856 bytes 2024-09-14T23:03:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2617707520 bytes 2024-09-14T23:03:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2649511936 bytes 2024-09-14T23:03:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2701714432 bytes 2024-09-14T23:03:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2757780480 bytes 2024-09-14T23:03:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2770533376 bytes 2024-09-14T23:03:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2819257344 bytes 2024-09-14T23:03:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2852931584 bytes 2024-09-14T23:03:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_16 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2024-09-14T23:03:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:03:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:03:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-14T23:03:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:03:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-14T23:03:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:03:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-14T23:03:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_16': 2024-09-14T23:03:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:03:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:03:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:03:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 131098e893b 2024-09-14T23:03:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.131098e893b 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:17 2024 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:17 2024 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:17 2024 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: ff85a14f-4301-4076-86de-a3f6e6b1ddd9 2024-09-14T23:03:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:03:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:03:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-14T23:03:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:03:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:03:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=10 2024-09-14T23:03:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:03:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:03:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 21022720 bytes 2024-09-14T23:03:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 41445376 bytes 2024-09-14T23:03:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 100445184 bytes 2024-09-14T23:03:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 116094976 bytes 2024-09-14T23:03:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 129771520 bytes 2024-09-14T23:03:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 187282432 bytes 2024-09-14T23:03:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 197518336 bytes 2024-09-14T23:03:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 198140928 bytes 2024-09-14T23:03:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 235780096 bytes 2024-09-14T23:03:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 249835520 bytes 2024-09-14T23:03:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 290061312 bytes 2024-09-14T23:03:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 325646336 bytes 2024-09-14T23:03:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 426999808 bytes 2024-09-14T23:03:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 472432640 bytes 2024-09-14T23:03:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 497017856 bytes 2024-09-14T23:03:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 550986752 bytes 2024-09-14T23:03:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 586969088 bytes 2024-09-14T23:03:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 605663232 bytes 2024-09-14T23:03:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 674340864 bytes 2024-09-14T23:03:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 747550720 bytes 2024-09-14T23:03:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 792833024 bytes 2024-09-14T23:03:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 840065024 bytes 2024-09-14T23:03:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 882845696 bytes 2024-09-14T23:03:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 939403264 bytes 2024-09-14T23:03:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 987986944 bytes 2024-09-14T23:03:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1044889600 bytes 2024-09-14T23:03:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1075388416 bytes 2024-09-14T23:03:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1139261440 bytes 2024-09-14T23:03:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1156102144 bytes 2024-09-14T23:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1167252480 bytes 2024-09-14T23:03:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1204954112 bytes 2024-09-14T23:03:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1210717184 bytes 2024-09-14T23:03:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1276216320 bytes 2024-09-14T23:03:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1326257152 bytes 2024-09-14T23:03:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1338569728 bytes 2024-09-14T23:03:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1393098752 bytes 2024-09-14T23:03:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1486596096 bytes 2024-09-14T23:03:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1528664064 bytes 2024-09-14T23:03:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1602438144 bytes 2024-09-14T23:03:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1634356224 bytes 2024-09-14T23:03:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1642569728 bytes 2024-09-14T23:03:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1690896384 bytes 2024-09-14T23:03:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1719300096 bytes 2024-09-14T23:03:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1773020160 bytes 2024-09-14T23:03:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1830796288 bytes 2024-09-14T23:03:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1867393024 bytes 2024-09-14T23:03:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1940569088 bytes 2024-09-14T23:03:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1962918912 bytes 2024-09-14T23:03:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1999895552 bytes 2024-09-14T23:03:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2036065280 bytes 2024-09-14T23:03:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2063865856 bytes 2024-09-14T23:03:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2115512320 bytes 2024-09-14T23:03:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2167939072 bytes 2024-09-14T23:03:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2183138304 bytes 2024-09-14T23:03:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2287683584 bytes 2024-09-14T23:03:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2372353024 bytes 2024-09-14T23:03:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2411395072 bytes 2024-09-14T23:03:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2461259776 bytes 2024-09-14T23:03:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2511964160 bytes 2024-09-14T23:03:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2537806848 bytes 2024-09-14T23:03:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2583586816 bytes 2024-09-14T23:03:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2613321728 bytes 2024-09-14T23:03:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2621591552 bytes 2024-09-14T23:03:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2642912256 bytes 2024-09-14T23:03:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2655246336 bytes 2024-09-14T23:03:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2691653632 bytes 2024-09-14T23:03:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2702974976 bytes 2024-09-14T23:03:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2755245056 bytes 2024-09-14T23:03:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2839071744 bytes 2024-09-14T23:03:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2849963008 bytes 2024-09-14T23:03:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2906375168 bytes 2024-09-14T23:03:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2961709056 bytes 2024-09-14T23:03:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3008459776 bytes 2024-09-14T23:03:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3071713280 bytes 2024-09-14T23:03:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3086612480 bytes 2024-09-14T23:03:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3168180224 bytes 2024-09-14T23:03:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3214570496 bytes 2024-09-14T23:03:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3283349504 bytes 2024-09-14T23:03:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3303262208 bytes 2024-09-14T23:03:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3342344192 bytes 2024-09-14T23:03:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3383053312 bytes 2024-09-14T23:03:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3429496832 bytes 2024-09-14T23:03:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3475964928 bytes 2024-09-14T23:03:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3554792448 bytes 2024-09-14T23:03:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3591170048 bytes 2024-09-14T23:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3641800704 bytes 2024-09-14T23:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:03:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_17 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-14T23:03:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:03:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-14T23:03:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_17': 2024-09-14T23:03:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:03:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:03:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1319c5ba2d67 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1319c5ba2d67 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:28 2024 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:28 2024 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:28 2024 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 839cf198-206a-4346-91c9-f4360063f32f 2024-09-14T23:03:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:03:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:03:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-14T23:03:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:03:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:03:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T23:03:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:03:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:03:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 24985600 bytes 2024-09-14T23:03:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 72777728 bytes 2024-09-14T23:03:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 143321088 bytes 2024-09-14T23:03:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 150307840 bytes 2024-09-14T23:03:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 174015488 bytes 2024-09-14T23:03:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 207316992 bytes 2024-09-14T23:03:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 225879040 bytes 2024-09-14T23:03:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 250562560 bytes 2024-09-14T23:03:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 258751488 bytes 2024-09-14T23:03:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 259382272 bytes 2024-09-14T23:03:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 315951104 bytes 2024-09-14T23:03:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 378580992 bytes 2024-09-14T23:03:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 422354944 bytes 2024-09-14T23:03:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 444559360 bytes 2024-09-14T23:03:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 497582080 bytes 2024-09-14T23:03:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 535585792 bytes 2024-09-14T23:03:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 575354880 bytes 2024-09-14T23:03:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 597236736 bytes 2024-09-14T23:03:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 647117824 bytes 2024-09-14T23:03:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 654663680 bytes 2024-09-14T23:03:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 694385664 bytes 2024-09-14T23:03:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 711638016 bytes 2024-09-14T23:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 728823808 bytes 2024-09-14T23:03:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 817760256 bytes 2024-09-14T23:03:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 833081344 bytes 2024-09-14T23:03:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 833676288 bytes 2024-09-14T23:03:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 886878208 bytes 2024-09-14T23:03:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 917218304 bytes 2024-09-14T23:03:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 944887808 bytes 2024-09-14T23:03:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1004359680 bytes 2024-09-14T23:03:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1075290112 bytes 2024-09-14T23:03:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1148917760 bytes 2024-09-14T23:03:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1175758848 bytes 2024-09-14T23:03:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1226412032 bytes 2024-09-14T23:03:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1292425216 bytes 2024-09-14T23:03:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1319552000 bytes 2024-09-14T23:03:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1337556992 bytes 2024-09-14T23:03:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1406659584 bytes 2024-09-14T23:03:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1457645568 bytes 2024-09-14T23:03:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1469978624 bytes 2024-09-14T23:03:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1470620672 bytes 2024-09-14T23:03:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1577413632 bytes 2024-09-14T23:03:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1602582528 bytes 2024-09-14T23:03:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1675424768 bytes 2024-09-14T23:03:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1705015296 bytes 2024-09-14T23:03:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1800603648 bytes 2024-09-14T23:03:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1859714048 bytes 2024-09-14T23:03:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1893453824 bytes 2024-09-14T23:03:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1936690176 bytes 2024-09-14T23:03:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2032670720 bytes 2024-09-14T23:03:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2055299072 bytes 2024-09-14T23:03:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2118834176 bytes 2024-09-14T23:03:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2146780160 bytes 2024-09-14T23:03:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:03:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:03:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:03:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:03:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_18 2024-09-14T23:03:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-14T23:03:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:03:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:03:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2024-09-14T23:03:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:03:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:03:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-14T23:03:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:03:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-14T23:03:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:03:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-14T23:03:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_18': 2024-09-14T23:03:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:03:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:03:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:03:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13224bd71f56 2024-09-14T23:03:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13224bd71f56 2024-09-14T23:03:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:03:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:03:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:03:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:03:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:35 2024 2024-09-14T23:03:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:35 2024 2024-09-14T23:03:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:35 2024 2024-09-14T23:03:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:03:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:03:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: d80a064e-80a3-476f-9ce2-c2191cc27f0b 2024-09-14T23:03:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:03:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:03:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-14T23:03:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:03:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:03:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=12 2024-09-14T23:03:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:03:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:03:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 42439680 bytes 2024-09-14T23:03:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 63897600 bytes 2024-09-14T23:03:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 92404736 bytes 2024-09-14T23:03:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 128095232 bytes 2024-09-14T23:03:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 199192576 bytes 2024-09-14T23:03:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 225309696 bytes 2024-09-14T23:03:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 225934336 bytes 2024-09-14T23:03:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 308484096 bytes 2024-09-14T23:03:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 346030080 bytes 2024-09-14T23:03:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 351631360 bytes 2024-09-14T23:03:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 381203456 bytes 2024-09-14T23:03:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 427451392 bytes 2024-09-14T23:03:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 507314176 bytes 2024-09-14T23:03:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 570886144 bytes 2024-09-14T23:03:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 571533312 bytes 2024-09-14T23:03:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 626142208 bytes 2024-09-14T23:03:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 659450880 bytes 2024-09-14T23:03:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 706430976 bytes 2024-09-14T23:03:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 710098944 bytes 2024-09-14T23:03:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 720814080 bytes 2024-09-14T23:03:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 762305536 bytes 2024-09-14T23:03:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 816388096 bytes 2024-09-14T23:03:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 850970624 bytes 2024-09-14T23:03:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 900210688 bytes 2024-09-14T23:03:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 966862848 bytes 2024-09-14T23:03:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 986250240 bytes 2024-09-14T23:03:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1009827840 bytes 2024-09-14T23:03:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1034855424 bytes 2024-09-14T23:03:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1082901504 bytes 2024-09-14T23:03:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1124924416 bytes 2024-09-14T23:03:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1214999552 bytes 2024-09-14T23:03:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1238310912 bytes 2024-09-14T23:03:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1258874880 bytes 2024-09-14T23:03:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1287064576 bytes 2024-09-14T23:03:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1363368960 bytes 2024-09-14T23:03:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1373749248 bytes 2024-09-14T23:03:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1407155200 bytes 2024-09-14T23:03:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1483603968 bytes 2024-09-14T23:03:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1484541952 bytes 2024-09-14T23:03:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1541301248 bytes 2024-09-14T23:03:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1560769536 bytes 2024-09-14T23:03:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1572793344 bytes 2024-09-14T23:03:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1582511104 bytes 2024-09-14T23:03:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1600901120 bytes 2024-09-14T23:03:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1649918976 bytes 2024-09-14T23:03:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1670231040 bytes 2024-09-14T23:03:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1701271552 bytes 2024-09-14T23:03:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1701892096 bytes 2024-09-14T23:03:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1761485824 bytes 2024-09-14T23:03:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1767608320 bytes 2024-09-14T23:03:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 1790992384 bytes 2024-09-14T23:03:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 1820580864 bytes 2024-09-14T23:03:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 1899554816 bytes 2024-09-14T23:03:42.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 1948029952 bytes 2024-09-14T23:03:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 1951917056 bytes 2024-09-14T23:03:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 1967059968 bytes 2024-09-14T23:03:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2068920320 bytes 2024-09-14T23:03:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2086649856 bytes 2024-09-14T23:03:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2177637376 bytes 2024-09-14T23:03:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2240326656 bytes 2024-09-14T23:03:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2279339008 bytes 2024-09-14T23:03:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2314512384 bytes 2024-09-14T23:03:43.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2346297344 bytes 2024-09-14T23:03:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2416393216 bytes 2024-09-14T23:03:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2426421248 bytes 2024-09-14T23:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2443828224 bytes 2024-09-14T23:03:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2507825152 bytes 2024-09-14T23:03:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2582206464 bytes 2024-09-14T23:03:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2613873664 bytes 2024-09-14T23:03:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2636124160 bytes 2024-09-14T23:03:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2640910336 bytes 2024-09-14T23:03:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2687488000 bytes 2024-09-14T23:03:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2715175936 bytes 2024-09-14T23:03:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2772669440 bytes 2024-09-14T23:03:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 2816657408 bytes 2024-09-14T23:03:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 2817268736 bytes 2024-09-14T23:03:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 2891859968 bytes 2024-09-14T23:03:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 2937555968 bytes 2024-09-14T23:03:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 2986958848 bytes 2024-09-14T23:03:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3020984320 bytes 2024-09-14T23:03:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3054504960 bytes 2024-09-14T23:03:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3084640256 bytes 2024-09-14T23:03:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3099061248 bytes 2024-09-14T23:03:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3170207744 bytes 2024-09-14T23:03:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3205278720 bytes 2024-09-14T23:03:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3248709632 bytes 2024-09-14T23:03:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3267842048 bytes 2024-09-14T23:03:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3332714496 bytes 2024-09-14T23:03:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3361586176 bytes 2024-09-14T23:03:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3408605184 bytes 2024-09-14T23:03:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3468453888 bytes 2024-09-14T23:03:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3503544320 bytes 2024-09-14T23:03:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3577192448 bytes 2024-09-14T23:03:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3582537728 bytes 2024-09-14T23:03:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 3642514432 bytes 2024-09-14T23:03:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 3693608960 bytes 2024-09-14T23:03:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 3700217856 bytes 2024-09-14T23:03:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 3740115968 bytes 2024-09-14T23:03:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 3785200640 bytes 2024-09-14T23:03:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 3850167296 bytes 2024-09-14T23:03:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 3910768640 bytes 2024-09-14T23:03:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 3953385472 bytes 2024-09-14T23:03:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 3988048896 bytes 2024-09-14T23:03:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4000095232 bytes 2024-09-14T23:03:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4011200512 bytes 2024-09-14T23:03:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4016914432 bytes 2024-09-14T23:03:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4056276992 bytes 2024-09-14T23:03:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4087648256 bytes 2024-09-14T23:03:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 4088259584 bytes 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_19 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:03:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:03:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2024-09-14T23:03:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:03:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:03:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-14T23:03:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:03:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-14T23:03:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:03:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-14T23:03:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_19': 2024-09-14T23:03:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:03:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:03:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:03:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 132b1ff74cec 2024-09-14T23:03:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.132b1ff74cec 2024-09-14T23:03:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:48 2024 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:48 2024 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:48 2024 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5b5b353a-9976-4517-9150-396dd4235407 2024-09-14T23:03:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:03:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:03:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-14T23:03:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:03:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:03:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=7 2024-09-14T23:03:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:03:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:03:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 16890880 bytes 2024-09-14T23:03:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 28089344 bytes 2024-09-14T23:03:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 101702656 bytes 2024-09-14T23:03:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 137980928 bytes 2024-09-14T23:03:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 178217984 bytes 2024-09-14T23:03:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 233608192 bytes 2024-09-14T23:03:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 267613184 bytes 2024-09-14T23:03:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 287890432 bytes 2024-09-14T23:03:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 351055872 bytes 2024-09-14T23:03:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 372563968 bytes 2024-09-14T23:03:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 421518336 bytes 2024-09-14T23:03:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 483817472 bytes 2024-09-14T23:03:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 531384320 bytes 2024-09-14T23:03:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 549843968 bytes 2024-09-14T23:03:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 611674112 bytes 2024-09-14T23:03:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 637972480 bytes 2024-09-14T23:03:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 669870080 bytes 2024-09-14T23:03:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 670484480 bytes 2024-09-14T23:03:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 704953344 bytes 2024-09-14T23:03:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 768915456 bytes 2024-09-14T23:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 799803392 bytes 2024-09-14T23:03:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 800431104 bytes 2024-09-14T23:03:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 846409728 bytes 2024-09-14T23:03:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 847009792 bytes 2024-09-14T23:03:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 900849664 bytes 2024-09-14T23:03:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 950422528 bytes 2024-09-14T23:03:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 996314112 bytes 2024-09-14T23:03:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1038107648 bytes 2024-09-14T23:03:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1104327680 bytes 2024-09-14T23:03:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1128259584 bytes 2024-09-14T23:03:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1163276288 bytes 2024-09-14T23:03:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1202989056 bytes 2024-09-14T23:03:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1295869952 bytes 2024-09-14T23:03:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1336410112 bytes 2024-09-14T23:03:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1344620544 bytes 2024-09-14T23:03:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1461769216 bytes 2024-09-14T23:03:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1490465792 bytes 2024-09-14T23:03:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1501434880 bytes 2024-09-14T23:03:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1538061312 bytes 2024-09-14T23:03:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1550036992 bytes 2024-09-14T23:03:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1572458496 bytes 2024-09-14T23:03:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1694597120 bytes 2024-09-14T23:03:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1734342656 bytes 2024-09-14T23:03:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1765349376 bytes 2024-09-14T23:03:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1765935104 bytes 2024-09-14T23:03:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1832259584 bytes 2024-09-14T23:03:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1894888448 bytes 2024-09-14T23:03:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1906468864 bytes 2024-09-14T23:03:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1996004352 bytes 2024-09-14T23:03:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2055148544 bytes 2024-09-14T23:03:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2106275840 bytes 2024-09-14T23:03:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2155981824 bytes 2024-09-14T23:03:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2210672640 bytes 2024-09-14T23:03:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2283396096 bytes 2024-09-14T23:03:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2363476992 bytes 2024-09-14T23:03:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2391363584 bytes 2024-09-14T23:03:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2425075712 bytes 2024-09-14T23:03:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2478956544 bytes 2024-09-14T23:03:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2521743360 bytes 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_20 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:03:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:03:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:03:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:03:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-14T23:03:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:03:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-14T23:03:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_20': 2024-09-14T23:03:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:03:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:03:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13339caa197c 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13339caa197c 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:56 2024 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:56 2024 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:56 2024 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: c471720c-4eb1-4c8a-a244-9687727d167b 2024-09-14T23:03:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:03:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:03:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-14T23:03:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:03:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:03:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:03:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:03:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=11 2024-09-14T23:03:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:03:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:03:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 31478784 bytes 2024-09-14T23:03:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 114615296 bytes 2024-09-14T23:03:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 154800128 bytes 2024-09-14T23:03:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 194385920 bytes 2024-09-14T23:03:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 206699520 bytes 2024-09-14T23:03:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 217949184 bytes 2024-09-14T23:03:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 249418752 bytes 2024-09-14T23:03:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 268951552 bytes 2024-09-14T23:03:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 340112384 bytes 2024-09-14T23:03:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 366223360 bytes 2024-09-14T23:03:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 401952768 bytes 2024-09-14T23:03:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 520095744 bytes 2024-09-14T23:03:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 573720576 bytes 2024-09-14T23:03:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 637450240 bytes 2024-09-14T23:03:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 689620992 bytes 2024-09-14T23:03:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 690253824 bytes 2024-09-14T23:03:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 766588928 bytes 2024-09-14T23:03:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 772089856 bytes 2024-09-14T23:03:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 792574976 bytes 2024-09-14T23:03:59.875 INFO:tasks.ceph.cluster2.osd.0.smithi089.stderr:2024-09-14T23:03:59.736+0000 7fc768a4a640 -1 reset not still connected to 0x564937b8f5f0 2024-09-14T23:03:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 800131072 bytes 2024-09-14T23:03:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 817397760 bytes 2024-09-14T23:03:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 861336576 bytes 2024-09-14T23:04:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 920646656 bytes 2024-09-14T23:04:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1009127424 bytes 2024-09-14T23:04:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1051507712 bytes 2024-09-14T23:04:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1123743744 bytes 2024-09-14T23:04:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1167203328 bytes 2024-09-14T23:04:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1259033600 bytes 2024-09-14T23:04:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1277550592 bytes 2024-09-14T23:04:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1302868992 bytes 2024-09-14T23:04:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1356485632 bytes 2024-09-14T23:04:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1412664320 bytes 2024-09-14T23:04:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1470364672 bytes 2024-09-14T23:04:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1497238528 bytes 2024-09-14T23:04:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1561797632 bytes 2024-09-14T23:04:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1574137856 bytes 2024-09-14T23:04:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1599079424 bytes 2024-09-14T23:04:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1673799680 bytes 2024-09-14T23:04:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1734625280 bytes 2024-09-14T23:04:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1804372992 bytes 2024-09-14T23:04:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1817971712 bytes 2024-09-14T23:04:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1843426304 bytes 2024-09-14T23:04:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1868898304 bytes 2024-09-14T23:04:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1877332992 bytes 2024-09-14T23:04:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1937805312 bytes 2024-09-14T23:04:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1963905024 bytes 2024-09-14T23:04:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1989642240 bytes 2024-09-14T23:04:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2055419904 bytes 2024-09-14T23:04:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2093895680 bytes 2024-09-14T23:04:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2177978368 bytes 2024-09-14T23:04:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2179099648 bytes 2024-09-14T23:04:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2242826240 bytes 2024-09-14T23:04:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2281370624 bytes 2024-09-14T23:04:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2342595584 bytes 2024-09-14T23:04:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2389992448 bytes 2024-09-14T23:04:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2416401408 bytes 2024-09-14T23:04:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2475563008 bytes 2024-09-14T23:04:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2512138240 bytes 2024-09-14T23:04:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2565253120 bytes 2024-09-14T23:04:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2565876736 bytes 2024-09-14T23:04:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2632215552 bytes 2024-09-14T23:04:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2691778560 bytes 2024-09-14T23:04:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2764474368 bytes 2024-09-14T23:04:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2805398528 bytes 2024-09-14T23:04:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2847561728 bytes 2024-09-14T23:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2861819904 bytes 2024-09-14T23:04:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2908943360 bytes 2024-09-14T23:04:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2909642752 bytes 2024-09-14T23:04:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2996503552 bytes 2024-09-14T23:04:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3019781120 bytes 2024-09-14T23:04:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3082095616 bytes 2024-09-14T23:04:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3114923008 bytes 2024-09-14T23:04:06.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3138449408 bytes 2024-09-14T23:04:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3187130368 bytes 2024-09-14T23:04:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3215628288 bytes 2024-09-14T23:04:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3260156928 bytes 2024-09-14T23:04:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3297298432 bytes 2024-09-14T23:04:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3354921984 bytes 2024-09-14T23:04:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3398904832 bytes 2024-09-14T23:04:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3478680576 bytes 2024-09-14T23:04:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3517498368 bytes 2024-09-14T23:04:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3560552448 bytes 2024-09-14T23:04:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3589269504 bytes 2024-09-14T23:04:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3634632704 bytes 2024-09-14T23:04:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3719187456 bytes 2024-09-14T23:04:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3747904512 bytes 2024-09-14T23:04:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3803613184 bytes 2024-09-14T23:04:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3912318976 bytes 2024-09-14T23:04:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3957582848 bytes 2024-09-14T23:04:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 4026877952 bytes 2024-09-14T23:04:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 4088224768 bytes 2024-09-14T23:04:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 4097980416 bytes 2024-09-14T23:04:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:04:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:04:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:04:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:04:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_21 2024-09-14T23:04:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-14T23:04:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:04:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:04:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:04:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:04:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2024-09-14T23:04:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:04:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:04:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-14T23:04:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:04:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-14T23:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:04:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_21': 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 133ccc645b7c 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.133ccc645b7c 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:04:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:04:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:08 2024 2024-09-14T23:04:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:08 2024 2024-09-14T23:04:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:08 2024 2024-09-14T23:04:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:04:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:04:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 67fd617c-03d8-493d-bbbb-1c4c9b42e7b8 2024-09-14T23:04:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:04:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:04:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-14T23:04:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:04:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:04:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=8 2024-09-14T23:04:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:04:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:04:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 27090944 bytes 2024-09-14T23:04:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 61081600 bytes 2024-09-14T23:04:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 134694912 bytes 2024-09-14T23:04:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 173917184 bytes 2024-09-14T23:04:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 219125760 bytes 2024-09-14T23:04:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 240532480 bytes 2024-09-14T23:04:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 334930944 bytes 2024-09-14T23:04:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 408924160 bytes 2024-09-14T23:04:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 440134656 bytes 2024-09-14T23:04:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 503157760 bytes 2024-09-14T23:04:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 597671936 bytes 2024-09-14T23:04:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 624296960 bytes 2024-09-14T23:04:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 699518976 bytes 2024-09-14T23:04:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 711771136 bytes 2024-09-14T23:04:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 735316992 bytes 2024-09-14T23:04:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 743787520 bytes 2024-09-14T23:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 764079104 bytes 2024-09-14T23:04:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 805186560 bytes 2024-09-14T23:04:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 805782528 bytes 2024-09-14T23:04:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 825809920 bytes 2024-09-14T23:04:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 833133568 bytes 2024-09-14T23:04:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 881408000 bytes 2024-09-14T23:04:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 927229952 bytes 2024-09-14T23:04:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 947013632 bytes 2024-09-14T23:04:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 947630080 bytes 2024-09-14T23:04:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 969801728 bytes 2024-09-14T23:04:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 996042752 bytes 2024-09-14T23:04:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1035789312 bytes 2024-09-14T23:04:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1059525632 bytes 2024-09-14T23:04:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1134222336 bytes 2024-09-14T23:04:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1150601216 bytes 2024-09-14T23:04:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1216310272 bytes 2024-09-14T23:04:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1258134528 bytes 2024-09-14T23:04:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1260361728 bytes 2024-09-14T23:04:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1317230592 bytes 2024-09-14T23:04:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1355155456 bytes 2024-09-14T23:04:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1385710592 bytes 2024-09-14T23:04:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1422858240 bytes 2024-09-14T23:04:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1471342592 bytes 2024-09-14T23:04:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1480668160 bytes 2024-09-14T23:04:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1521930240 bytes 2024-09-14T23:04:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1628462080 bytes 2024-09-14T23:04:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1661734912 bytes 2024-09-14T23:04:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1713758208 bytes 2024-09-14T23:04:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1743411200 bytes 2024-09-14T23:04:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1774831616 bytes 2024-09-14T23:04:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1864259584 bytes 2024-09-14T23:04:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1905565696 bytes 2024-09-14T23:04:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1958950912 bytes 2024-09-14T23:04:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1978209280 bytes 2024-09-14T23:04:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2007800832 bytes 2024-09-14T23:04:15.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2057542656 bytes 2024-09-14T23:04:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2107325440 bytes 2024-09-14T23:04:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2124932096 bytes 2024-09-14T23:04:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2181214208 bytes 2024-09-14T23:04:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2216058880 bytes 2024-09-14T23:04:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2247689216 bytes 2024-09-14T23:04:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2315453440 bytes 2024-09-14T23:04:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2371831808 bytes 2024-09-14T23:04:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2385988608 bytes 2024-09-14T23:04:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2473219072 bytes 2024-09-14T23:04:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2498560000 bytes 2024-09-14T23:04:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2526350336 bytes 2024-09-14T23:04:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2551354368 bytes 2024-09-14T23:04:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2621296640 bytes 2024-09-14T23:04:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2624657408 bytes 2024-09-14T23:04:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2650172416 bytes 2024-09-14T23:04:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2696582144 bytes 2024-09-14T23:04:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2710573056 bytes 2024-09-14T23:04:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2763269120 bytes 2024-09-14T23:04:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_22 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:04:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:04:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-14T23:04:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:04:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:04:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:04:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:04:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2024-09-14T23:04:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:04:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:04:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-14T23:04:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:04:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-14T23:04:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:04:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-14T23:04:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_22': 2024-09-14T23:04:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:04:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:04:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:04:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 134579e729a7 2024-09-14T23:04:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.134579e729a7 2024-09-14T23:04:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:04:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:04:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:04:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:04:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:17 2024 2024-09-14T23:04:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:17 2024 2024-09-14T23:04:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:17 2024 2024-09-14T23:04:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:04:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:04:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: a9dd159e-4591-405f-b0f6-42ca7c8cdf26 2024-09-14T23:04:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:04:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:04:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-14T23:04:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:04:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:04:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=8 2024-09-14T23:04:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:04:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:04:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 25626624 bytes 2024-09-14T23:04:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 78539776 bytes 2024-09-14T23:04:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 131068928 bytes 2024-09-14T23:04:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 208824320 bytes 2024-09-14T23:04:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 267678720 bytes 2024-09-14T23:04:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 288250880 bytes 2024-09-14T23:04:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 331677696 bytes 2024-09-14T23:04:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 379729920 bytes 2024-09-14T23:04:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 443687936 bytes 2024-09-14T23:04:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 468594688 bytes 2024-09-14T23:04:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 529918976 bytes 2024-09-14T23:04:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 554685440 bytes 2024-09-14T23:04:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 596822016 bytes 2024-09-14T23:04:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 644632576 bytes 2024-09-14T23:04:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 685517824 bytes 2024-09-14T23:04:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 725848064 bytes 2024-09-14T23:04:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 755453952 bytes 2024-09-14T23:04:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 811253760 bytes 2024-09-14T23:04:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 848198656 bytes 2024-09-14T23:04:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 942430208 bytes 2024-09-14T23:04:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 1002339328 bytes 2024-09-14T23:04:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1036752896 bytes 2024-09-14T23:04:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1063628800 bytes 2024-09-14T23:04:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1158185984 bytes 2024-09-14T23:04:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1211074560 bytes 2024-09-14T23:04:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1222246400 bytes 2024-09-14T23:04:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1273799680 bytes 2024-09-14T23:04:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1340217344 bytes 2024-09-14T23:04:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1350394880 bytes 2024-09-14T23:04:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1398506496 bytes 2024-09-14T23:04:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1428654080 bytes 2024-09-14T23:04:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1471459328 bytes 2024-09-14T23:04:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1496851456 bytes 2024-09-14T23:04:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1516692480 bytes 2024-09-14T23:04:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1649891328 bytes 2024-09-14T23:04:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1656095744 bytes 2024-09-14T23:04:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1681908736 bytes 2024-09-14T23:04:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1729453056 bytes 2024-09-14T23:04:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1741616128 bytes 2024-09-14T23:04:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1801474048 bytes 2024-09-14T23:04:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1824374784 bytes 2024-09-14T23:04:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1837987840 bytes 2024-09-14T23:04:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1855156224 bytes 2024-09-14T23:04:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1921833984 bytes 2024-09-14T23:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1956797440 bytes 2024-09-14T23:04:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2094253056 bytes 2024-09-14T23:04:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2124020736 bytes 2024-09-14T23:04:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2164261888 bytes 2024-09-14T23:04:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2246161408 bytes 2024-09-14T23:04:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2285794304 bytes 2024-09-14T23:04:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2307280896 bytes 2024-09-14T23:04:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2332892160 bytes 2024-09-14T23:04:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2342064128 bytes 2024-09-14T23:04:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2372464640 bytes 2024-09-14T23:04:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2431830016 bytes 2024-09-14T23:04:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2499093504 bytes 2024-09-14T23:04:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2550377472 bytes 2024-09-14T23:04:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2582927360 bytes 2024-09-14T23:04:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2719240192 bytes 2024-09-14T23:04:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2782077952 bytes 2024-09-14T23:04:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2856841216 bytes 2024-09-14T23:04:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2937596928 bytes 2024-09-14T23:04:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 3001787392 bytes 2024-09-14T23:04:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_23 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:04:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2024-09-14T23:04:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:04:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:04:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-14T23:04:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:04:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-14T23:04:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:04:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-14T23:04:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_23': 2024-09-14T23:04:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 134dc3c1a063 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.134dc3c1a063 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:26 2024 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:26 2024 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:26 2024 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:04:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: f42fac99-ab54-44f2-ac18-c845cd99d02b 2024-09-14T23:04:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:04:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:04:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-14T23:04:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:04:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T23:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:04:27.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 50995200 bytes 2024-09-14T23:04:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 89681920 bytes 2024-09-14T23:04:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 136011776 bytes 2024-09-14T23:04:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 184306688 bytes 2024-09-14T23:04:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 184894464 bytes 2024-09-14T23:04:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 228363264 bytes 2024-09-14T23:04:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 246496256 bytes 2024-09-14T23:04:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 274249728 bytes 2024-09-14T23:04:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 274906112 bytes 2024-09-14T23:04:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 324313088 bytes 2024-09-14T23:04:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 397368320 bytes 2024-09-14T23:04:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 425936896 bytes 2024-09-14T23:04:28.905 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.980964385325192) > weight (0.25) 2024-09-14T23:04:28.905 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2798401494391377) > weight (0.25) 2024-09-14T23:04:28.905 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-14T23:04:28.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-14T23:04:28.906 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-14T23:04:28.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:04:28.905+0000 7fd595355640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 46013) UID: 1000 2024-09-14T23:04:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 451552256 bytes 2024-09-14T23:04:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 501371904 bytes 2024-09-14T23:04:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 535769088 bytes 2024-09-14T23:04:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 622053376 bytes 2024-09-14T23:04:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 622680064 bytes 2024-09-14T23:04:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 645038080 bytes 2024-09-14T23:04:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 690249728 bytes 2024-09-14T23:04:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 743482368 bytes 2024-09-14T23:04:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 762634240 bytes 2024-09-14T23:04:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 844573696 bytes 2024-09-14T23:04:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 945452032 bytes 2024-09-14T23:04:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 947850240 bytes 2024-09-14T23:04:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1061967872 bytes 2024-09-14T23:04:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1125076992 bytes 2024-09-14T23:04:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1125686272 bytes 2024-09-14T23:04:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1171409920 bytes 2024-09-14T23:04:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1226190848 bytes 2024-09-14T23:04:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1254978560 bytes 2024-09-14T23:04:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1360173056 bytes 2024-09-14T23:04:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1371319296 bytes 2024-09-14T23:04:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1401530368 bytes 2024-09-14T23:04:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1433240576 bytes 2024-09-14T23:04:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1448642560 bytes 2024-09-14T23:04:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1514466304 bytes 2024-09-14T23:04:31.907 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-14T23:04:31.907 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:04:31.908 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-14T23:04:31.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T23:04:31.908 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T23:04:31.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T23:04:31.911 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-14T23:04:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1577476096 bytes 2024-09-14T23:04:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1640706048 bytes 2024-09-14T23:04:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1680519168 bytes 2024-09-14T23:04:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1743083520 bytes 2024-09-14T23:04:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1765519360 bytes 2024-09-14T23:04:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1833293824 bytes 2024-09-14T23:04:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1935656960 bytes 2024-09-14T23:04:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 2017414144 bytes 2024-09-14T23:04:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2056114176 bytes 2024-09-14T23:04:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2151798784 bytes 2024-09-14T23:04:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2152391680 bytes 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_24 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:04:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:04:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-14T23:04:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:04:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-14T23:04:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_24': 2024-09-14T23:04:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:04:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:04:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:04:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1360a08bf127 2024-09-14T23:04:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1360a08bf127 2024-09-14T23:04:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:04:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:33 2024 2024-09-14T23:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:33 2024 2024-09-14T23:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:33 2024 2024-09-14T23:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: d02fb2d0-2bf7-49a8-86d4-ecd3c3240c86 2024-09-14T23:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:04:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:04:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-14T23:04:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:04:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:04:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=5 2024-09-14T23:04:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:04:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:04:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 23408640 bytes 2024-09-14T23:04:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 52812800 bytes 2024-09-14T23:04:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 60634112 bytes 2024-09-14T23:04:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 61281280 bytes 2024-09-14T23:04:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 101647360 bytes 2024-09-14T23:04:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 149124096 bytes 2024-09-14T23:04:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 171554816 bytes 2024-09-14T23:04:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 198746112 bytes 2024-09-14T23:04:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 249446400 bytes 2024-09-14T23:04:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 276569088 bytes 2024-09-14T23:04:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 299901952 bytes 2024-09-14T23:04:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 337620992 bytes 2024-09-14T23:04:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 397762560 bytes 2024-09-14T23:04:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 439276544 bytes 2024-09-14T23:04:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 473282560 bytes 2024-09-14T23:04:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 515772416 bytes 2024-09-14T23:04:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 534133760 bytes 2024-09-14T23:04:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 553733120 bytes 2024-09-14T23:04:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 594992128 bytes 2024-09-14T23:04:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 629177344 bytes 2024-09-14T23:04:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 703349760 bytes 2024-09-14T23:04:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 735969280 bytes 2024-09-14T23:04:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 782301184 bytes 2024-09-14T23:04:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 813372416 bytes 2024-09-14T23:04:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 910624768 bytes 2024-09-14T23:04:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 984933376 bytes 2024-09-14T23:04:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 990911488 bytes 2024-09-14T23:04:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1030227968 bytes 2024-09-14T23:04:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1070519296 bytes 2024-09-14T23:04:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1091635200 bytes 2024-09-14T23:04:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1179534336 bytes 2024-09-14T23:04:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1191127040 bytes 2024-09-14T23:04:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1269265408 bytes 2024-09-14T23:04:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1306002432 bytes 2024-09-14T23:04:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1316764672 bytes 2024-09-14T23:04:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1321610240 bytes 2024-09-14T23:04:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1333279744 bytes 2024-09-14T23:04:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1380648960 bytes 2024-09-14T23:04:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1431240704 bytes 2024-09-14T23:04:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1573244928 bytes 2024-09-14T23:04:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1628295168 bytes 2024-09-14T23:04:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1692001280 bytes 2024-09-14T23:04:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:04:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:04:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:04:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:04:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_25 2024-09-14T23:04:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-14T23:04:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:04:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:04:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:04:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:04:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2024-09-14T23:04:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:04:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:04:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-14T23:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-14T23:04:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:04:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-14T23:04:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_25': 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13692d837f30 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13692d837f30 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:39 2024 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:39 2024 2024-09-14T23:04:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:39 2024 2024-09-14T23:04:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:04:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:04:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 16ceb1c6-1a5c-4b7b-8b20-da6200f477b6 2024-09-14T23:04:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:04:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:04:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-14T23:04:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:04:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:04:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=14 2024-09-14T23:04:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:04:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:04:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 12858368 bytes 2024-09-14T23:04:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 30900224 bytes 2024-09-14T23:04:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 92993536 bytes 2024-09-14T23:04:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 112120832 bytes 2024-09-14T23:04:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 175704064 bytes 2024-09-14T23:04:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 233351168 bytes 2024-09-14T23:04:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 322806784 bytes 2024-09-14T23:04:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 403972096 bytes 2024-09-14T23:04:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 461407232 bytes 2024-09-14T23:04:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 519631872 bytes 2024-09-14T23:04:42.121 INFO:tasks.ceph.cluster2.osd.0.smithi089.stderr:2024-09-14T23:04:42.118+0000 7fc768a4a640 -1 reset not still connected to 0x564924d18b60 2024-09-14T23:04:42.121 INFO:tasks.ceph.cluster2.osd.0.smithi089.stderr:2024-09-14T23:04:42.118+0000 7fc768a4a640 -1 reset not still connected to 0x56493d455790 2024-09-14T23:04:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 587004928 bytes 2024-09-14T23:04:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 622384128 bytes 2024-09-14T23:04:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 668237824 bytes 2024-09-14T23:04:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 698085376 bytes 2024-09-14T23:04:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 740475904 bytes 2024-09-14T23:04:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 802770944 bytes 2024-09-14T23:04:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 808376320 bytes 2024-09-14T23:04:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 814014464 bytes 2024-09-14T23:04:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 862800896 bytes 2024-09-14T23:04:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 895071232 bytes 2024-09-14T23:04:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 979333120 bytes 2024-09-14T23:04:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 979942400 bytes 2024-09-14T23:04:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1006151680 bytes 2024-09-14T23:04:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1028658176 bytes 2024-09-14T23:04:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1050491904 bytes 2024-09-14T23:04:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1116139520 bytes 2024-09-14T23:04:44.014 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6789984703016742) > weight (0.25) 2024-09-14T23:04:44.014 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8412011293940952) > weight (0.25) 2024-09-14T23:04:44.014 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9226258769791649) > weight (0.25) 2024-09-14T23:04:44.014 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4991660159219987) > weight (0.25) 2024-09-14T23:04:44.014 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-09-14T23:04:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1185613824 bytes 2024-09-14T23:04:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1197512704 bytes 2024-09-14T23:04:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1243686912 bytes 2024-09-14T23:04:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1269414912 bytes 2024-09-14T23:04:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1306000384 bytes 2024-09-14T23:04:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1332595712 bytes 2024-09-14T23:04:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1420876800 bytes 2024-09-14T23:04:44.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1425503232 bytes 2024-09-14T23:04:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1524285440 bytes 2024-09-14T23:04:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1590510592 bytes 2024-09-14T23:04:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1638192128 bytes 2024-09-14T23:04:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1648678912 bytes 2024-09-14T23:04:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1693560832 bytes 2024-09-14T23:04:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1759661056 bytes 2024-09-14T23:04:45.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1793171456 bytes 2024-09-14T23:04:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1858652160 bytes 2024-09-14T23:04:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1874646016 bytes 2024-09-14T23:04:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1923534848 bytes 2024-09-14T23:04:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1985289216 bytes 2024-09-14T23:04:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2015502336 bytes 2024-09-14T23:04:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2090339328 bytes 2024-09-14T23:04:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2125224960 bytes 2024-09-14T23:04:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2207193088 bytes 2024-09-14T23:04:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2262597632 bytes 2024-09-14T23:04:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2279274496 bytes 2024-09-14T23:04:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2303834112 bytes 2024-09-14T23:04:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2349041664 bytes 2024-09-14T23:04:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2350950400 bytes 2024-09-14T23:04:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2401416192 bytes 2024-09-14T23:04:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2479978496 bytes 2024-09-14T23:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2502284288 bytes 2024-09-14T23:04:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2550128640 bytes 2024-09-14T23:04:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2634236928 bytes 2024-09-14T23:04:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2666006528 bytes 2024-09-14T23:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2720377856 bytes 2024-09-14T23:04:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2726732800 bytes 2024-09-14T23:04:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2751213568 bytes 2024-09-14T23:04:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2777241600 bytes 2024-09-14T23:04:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2812231680 bytes 2024-09-14T23:04:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2834290688 bytes 2024-09-14T23:04:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2904163328 bytes 2024-09-14T23:04:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2950234112 bytes 2024-09-14T23:04:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2980393984 bytes 2024-09-14T23:04:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3001203712 bytes 2024-09-14T23:04:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3070858240 bytes 2024-09-14T23:04:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3103980544 bytes 2024-09-14T23:04:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3192184832 bytes 2024-09-14T23:04:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3192820736 bytes 2024-09-14T23:04:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3215187968 bytes 2024-09-14T23:04:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3247330304 bytes 2024-09-14T23:04:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3255406592 bytes 2024-09-14T23:04:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3281594368 bytes 2024-09-14T23:04:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3340471296 bytes 2024-09-14T23:04:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3454492672 bytes 2024-09-14T23:04:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3504907264 bytes 2024-09-14T23:04:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3537082368 bytes 2024-09-14T23:04:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3579783168 bytes 2024-09-14T23:04:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3604393984 bytes 2024-09-14T23:04:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3654191104 bytes 2024-09-14T23:04:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3751497728 bytes 2024-09-14T23:04:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3825852416 bytes 2024-09-14T23:04:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3858019328 bytes 2024-09-14T23:04:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3874756608 bytes 2024-09-14T23:04:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3898027008 bytes 2024-09-14T23:04:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3930186752 bytes 2024-09-14T23:04:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3984098304 bytes 2024-09-14T23:04:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 4002254848 bytes 2024-09-14T23:04:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4033774592 bytes 2024-09-14T23:04:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4093544448 bytes 2024-09-14T23:04:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4231695360 bytes 2024-09-14T23:04:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4269325312 bytes 2024-09-14T23:04:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4307302400 bytes 2024-09-14T23:04:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4389470208 bytes 2024-09-14T23:04:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4467673088 bytes 2024-09-14T23:04:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4484579328 bytes 2024-09-14T23:04:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4531063808 bytes 2024-09-14T23:04:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4538304512 bytes 2024-09-14T23:04:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4614543360 bytes 2024-09-14T23:04:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4654306304 bytes 2024-09-14T23:04:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4712117248 bytes 2024-09-14T23:04:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4742261760 bytes 2024-09-14T23:04:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4742848512 bytes 2024-09-14T23:04:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 4798638080 bytes 2024-09-14T23:04:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11000 IOs, 4892188672 bytes 2024-09-14T23:04:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11100 IOs, 4940058624 bytes 2024-09-14T23:04:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11200 IOs, 4984741888 bytes 2024-09-14T23:04:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11300 IOs, 5084488704 bytes 2024-09-14T23:04:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:04:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:04:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:04:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_26 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:04:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-14T23:04:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:04:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:04:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:04:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:04:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:04:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2024-09-14T23:04:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:04:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:04:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:04:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-14T23:04:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:04:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:04:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-14T23:04:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:04:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_26': 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1372b4d1c7b7 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1372b4d1c7b7 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:04:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:54 2024 2024-09-14T23:04:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:54 2024 2024-09-14T23:04:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:54 2024 2024-09-14T23:04:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:04:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:04:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 68d375d5-545a-4f5c-87a8-2ef5ae048705 2024-09-14T23:04:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:04:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:04:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-14T23:04:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:04:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:04:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:04:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:04:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=5 2024-09-14T23:04:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:04:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:04:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 25294848 bytes 2024-09-14T23:04:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 26418176 bytes 2024-09-14T23:04:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 41338880 bytes 2024-09-14T23:04:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 71335936 bytes 2024-09-14T23:04:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 112246784 bytes 2024-09-14T23:04:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 198310912 bytes 2024-09-14T23:04:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 222529536 bytes 2024-09-14T23:04:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 307540992 bytes 2024-09-14T23:04:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 376119296 bytes 2024-09-14T23:04:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 424843264 bytes 2024-09-14T23:04:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 458667008 bytes 2024-09-14T23:04:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 482145280 bytes 2024-09-14T23:04:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 485993472 bytes 2024-09-14T23:04:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 536131584 bytes 2024-09-14T23:04:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 566635520 bytes 2024-09-14T23:04:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 661335040 bytes 2024-09-14T23:04:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 661964800 bytes 2024-09-14T23:04:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 682941440 bytes 2024-09-14T23:04:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 720563200 bytes 2024-09-14T23:04:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 774018048 bytes 2024-09-14T23:04:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 788642816 bytes 2024-09-14T23:04:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 863590400 bytes 2024-09-14T23:04:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 911579136 bytes 2024-09-14T23:04:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 975343616 bytes 2024-09-14T23:04:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1021874176 bytes 2024-09-14T23:04:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1022505984 bytes 2024-09-14T23:04:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1091017728 bytes 2024-09-14T23:04:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1133514752 bytes 2024-09-14T23:04:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1172400128 bytes 2024-09-14T23:04:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1276225536 bytes 2024-09-14T23:04:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1295193088 bytes 2024-09-14T23:04:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1333568512 bytes 2024-09-14T23:04:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1373985792 bytes 2024-09-14T23:04:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1382235136 bytes 2024-09-14T23:04:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1439955968 bytes 2024-09-14T23:04:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1486747648 bytes 2024-09-14T23:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1497974784 bytes 2024-09-14T23:05:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1583497216 bytes 2024-09-14T23:05:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1613919232 bytes 2024-09-14T23:05:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1614906368 bytes 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_27 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:05:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:05:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:05:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2024-09-14T23:05:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:05:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:05:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-14T23:05:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:05:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-14T23:05:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:05:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-14T23:05:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_27': 2024-09-14T23:05:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:05:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:05:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 137b6c8ed7cf 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.137b6c8ed7cf 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:00 2024 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:00 2024 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:00 2024 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: a848ce88-5df8-4b90-b6a1-72b3261b72bc 2024-09-14T23:05:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:05:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:05:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-14T23:05:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:05:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:05:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=14 2024-09-14T23:05:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:05:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:05:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 72267776 bytes 2024-09-14T23:05:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 187820032 bytes 2024-09-14T23:05:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 210293760 bytes 2024-09-14T23:05:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 261004288 bytes 2024-09-14T23:05:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 387044352 bytes 2024-09-14T23:05:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 395020288 bytes 2024-09-14T23:05:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 463333376 bytes 2024-09-14T23:05:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 483200000 bytes 2024-09-14T23:05:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 546385920 bytes 2024-09-14T23:05:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 600351744 bytes 2024-09-14T23:05:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 651430912 bytes 2024-09-14T23:05:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 676973568 bytes 2024-09-14T23:05:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 777863168 bytes 2024-09-14T23:05:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 806367232 bytes 2024-09-14T23:05:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 835027968 bytes 2024-09-14T23:05:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 859908096 bytes 2024-09-14T23:05:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 889266176 bytes 2024-09-14T23:05:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 922007552 bytes 2024-09-14T23:05:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 1012312064 bytes 2024-09-14T23:05:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 1027916800 bytes 2024-09-14T23:05:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 1056757760 bytes 2024-09-14T23:05:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1071852544 bytes 2024-09-14T23:05:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1118106624 bytes 2024-09-14T23:05:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1205410816 bytes 2024-09-14T23:05:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1212519424 bytes 2024-09-14T23:05:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1251881984 bytes 2024-09-14T23:05:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1272817664 bytes 2024-09-14T23:05:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1296044032 bytes 2024-09-14T23:05:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1323627520 bytes 2024-09-14T23:05:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1347789824 bytes 2024-09-14T23:05:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1351356416 bytes 2024-09-14T23:05:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1386173440 bytes 2024-09-14T23:05:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1437968384 bytes 2024-09-14T23:05:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1458685952 bytes 2024-09-14T23:05:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1500814336 bytes 2024-09-14T23:05:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1599055872 bytes 2024-09-14T23:05:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1647238144 bytes 2024-09-14T23:05:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1666034688 bytes 2024-09-14T23:05:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1692773376 bytes 2024-09-14T23:05:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1723008000 bytes 2024-09-14T23:05:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1751315456 bytes 2024-09-14T23:05:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1775266816 bytes 2024-09-14T23:05:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1811065856 bytes 2024-09-14T23:05:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1828720640 bytes 2024-09-14T23:05:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1916328960 bytes 2024-09-14T23:05:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1961639936 bytes 2024-09-14T23:05:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1986638848 bytes 2024-09-14T23:05:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2024467456 bytes 2024-09-14T23:05:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2076936192 bytes 2024-09-14T23:05:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2106089472 bytes 2024-09-14T23:05:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2116623360 bytes 2024-09-14T23:05:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2186415104 bytes 2024-09-14T23:05:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2220192768 bytes 2024-09-14T23:05:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2274825216 bytes 2024-09-14T23:05:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2321728512 bytes 2024-09-14T23:05:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2330655744 bytes 2024-09-14T23:05:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2383057920 bytes 2024-09-14T23:05:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2402995200 bytes 2024-09-14T23:05:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2406808576 bytes 2024-09-14T23:05:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2415874048 bytes 2024-09-14T23:05:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2436892672 bytes 2024-09-14T23:05:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2519239680 bytes 2024-09-14T23:05:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2584263680 bytes 2024-09-14T23:05:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2610496512 bytes 2024-09-14T23:05:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2623522816 bytes 2024-09-14T23:05:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2696322048 bytes 2024-09-14T23:05:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2725433344 bytes 2024-09-14T23:05:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2761992192 bytes 2024-09-14T23:05:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2834442240 bytes 2024-09-14T23:05:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2835053568 bytes 2024-09-14T23:05:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2878550016 bytes 2024-09-14T23:05:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2926865408 bytes 2024-09-14T23:05:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2984263680 bytes 2024-09-14T23:05:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3004709888 bytes 2024-09-14T23:05:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3037004800 bytes 2024-09-14T23:05:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3110228992 bytes 2024-09-14T23:05:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3200839680 bytes 2024-09-14T23:05:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3244701696 bytes 2024-09-14T23:05:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3267796992 bytes 2024-09-14T23:05:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3326853120 bytes 2024-09-14T23:05:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3393270784 bytes 2024-09-14T23:05:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3415996416 bytes 2024-09-14T23:05:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3489487872 bytes 2024-09-14T23:05:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3519872000 bytes 2024-09-14T23:05:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3559622656 bytes 2024-09-14T23:05:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3586583552 bytes 2024-09-14T23:05:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3643583488 bytes 2024-09-14T23:05:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3672981504 bytes 2024-09-14T23:05:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3685334016 bytes 2024-09-14T23:05:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3689775104 bytes 2024-09-14T23:05:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3723015168 bytes 2024-09-14T23:05:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3772200960 bytes 2024-09-14T23:05:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3797051392 bytes 2024-09-14T23:05:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3797680128 bytes 2024-09-14T23:05:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 3892412416 bytes 2024-09-14T23:05:13.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 3930828800 bytes 2024-09-14T23:05:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 3984496640 bytes 2024-09-14T23:05:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4002021376 bytes 2024-09-14T23:05:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4058656768 bytes 2024-09-14T23:05:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4154098688 bytes 2024-09-14T23:05:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4198977536 bytes 2024-09-14T23:05:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4241401856 bytes 2024-09-14T23:05:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4277996544 bytes 2024-09-14T23:05:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4318909440 bytes 2024-09-14T23:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4340579328 bytes 2024-09-14T23:05:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4341432320 bytes 2024-09-14T23:05:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4343662592 bytes 2024-09-14T23:05:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4378436608 bytes 2024-09-14T23:05:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 4429807616 bytes 2024-09-14T23:05:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11000 IOs, 4470394880 bytes 2024-09-14T23:05:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11100 IOs, 4492718080 bytes 2024-09-14T23:05:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11200 IOs, 4546975744 bytes 2024-09-14T23:05:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11300 IOs, 4571243520 bytes 2024-09-14T23:05:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11400 IOs, 4617496576 bytes 2024-09-14T23:05:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11500 IOs, 4698278912 bytes 2024-09-14T23:05:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11600 IOs, 4761291776 bytes 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_28 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:05:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2024-09-14T23:05:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:05:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:05:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-14T23:05:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:05:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-14T23:05:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:05:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-14T23:05:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_28': 2024-09-14T23:05:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:05:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 138391a666c4 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.138391a666c4 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:15 2024 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:15 2024 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:15 2024 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 4c9c9805-ba1a-48da-a08b-8de70de557b3 2024-09-14T23:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:05:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:05:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-14T23:05:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:05:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:05:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=14 2024-09-14T23:05:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:05:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:05:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 5584896 bytes 2024-09-14T23:05:16.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 81978368 bytes 2024-09-14T23:05:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 133026816 bytes 2024-09-14T23:05:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 158899200 bytes 2024-09-14T23:05:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 217632768 bytes 2024-09-14T23:05:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 280173568 bytes 2024-09-14T23:05:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 320881664 bytes 2024-09-14T23:05:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 369221632 bytes 2024-09-14T23:05:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 393406464 bytes 2024-09-14T23:05:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 395981824 bytes 2024-09-14T23:05:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 476255232 bytes 2024-09-14T23:05:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 488952832 bytes 2024-09-14T23:05:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 505223168 bytes 2024-09-14T23:05:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 505843712 bytes 2024-09-14T23:05:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 557431808 bytes 2024-09-14T23:05:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 582130688 bytes 2024-09-14T23:05:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 639796224 bytes 2024-09-14T23:05:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 676381696 bytes 2024-09-14T23:05:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 739447808 bytes 2024-09-14T23:05:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 795850752 bytes 2024-09-14T23:05:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 827884544 bytes 2024-09-14T23:05:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 876583936 bytes 2024-09-14T23:05:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 918016000 bytes 2024-09-14T23:05:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 946575360 bytes 2024-09-14T23:05:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 992264192 bytes 2024-09-14T23:05:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1033236480 bytes 2024-09-14T23:05:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1092375552 bytes 2024-09-14T23:05:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1100047360 bytes 2024-09-14T23:05:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1146280960 bytes 2024-09-14T23:05:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1165881344 bytes 2024-09-14T23:05:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1171423232 bytes 2024-09-14T23:05:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1221191680 bytes 2024-09-14T23:05:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1259923456 bytes 2024-09-14T23:05:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1322717184 bytes 2024-09-14T23:05:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1371795456 bytes 2024-09-14T23:05:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1469883392 bytes 2024-09-14T23:05:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1529534464 bytes 2024-09-14T23:05:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1577223168 bytes 2024-09-14T23:05:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1597519872 bytes 2024-09-14T23:05:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1695772672 bytes 2024-09-14T23:05:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1696386048 bytes 2024-09-14T23:05:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1764994048 bytes 2024-09-14T23:05:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1836024832 bytes 2024-09-14T23:05:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1876887552 bytes 2024-09-14T23:05:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1892326400 bytes 2024-09-14T23:05:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1991634944 bytes 2024-09-14T23:05:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2062336000 bytes 2024-09-14T23:05:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2176662528 bytes 2024-09-14T23:05:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2231674880 bytes 2024-09-14T23:05:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2315816960 bytes 2024-09-14T23:05:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2316426240 bytes 2024-09-14T23:05:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2359403520 bytes 2024-09-14T23:05:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2439200768 bytes 2024-09-14T23:05:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2463429632 bytes 2024-09-14T23:05:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2507600896 bytes 2024-09-14T23:05:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2538136576 bytes 2024-09-14T23:05:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2613010432 bytes 2024-09-14T23:05:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2625722368 bytes 2024-09-14T23:05:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2698148864 bytes 2024-09-14T23:05:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2761651200 bytes 2024-09-14T23:05:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2795344896 bytes 2024-09-14T23:05:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2842728448 bytes 2024-09-14T23:05:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2885886976 bytes 2024-09-14T23:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2904727552 bytes 2024-09-14T23:05:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2908815360 bytes 2024-09-14T23:05:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2960737280 bytes 2024-09-14T23:05:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2976352256 bytes 2024-09-14T23:05:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2980819968 bytes 2024-09-14T23:05:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3064828928 bytes 2024-09-14T23:05:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3072010240 bytes 2024-09-14T23:05:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3162841088 bytes 2024-09-14T23:05:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3224969216 bytes 2024-09-14T23:05:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3259623424 bytes 2024-09-14T23:05:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3308043264 bytes 2024-09-14T23:05:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3308653568 bytes 2024-09-14T23:05:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3372348416 bytes 2024-09-14T23:05:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3391592448 bytes 2024-09-14T23:05:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3411827712 bytes 2024-09-14T23:05:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3449952256 bytes 2024-09-14T23:05:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3487155200 bytes 2024-09-14T23:05:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3532264448 bytes 2024-09-14T23:05:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3565240320 bytes 2024-09-14T23:05:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3619903488 bytes 2024-09-14T23:05:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3634626560 bytes 2024-09-14T23:05:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3690957824 bytes 2024-09-14T23:05:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3740878848 bytes 2024-09-14T23:05:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3792064512 bytes 2024-09-14T23:05:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3824192512 bytes 2024-09-14T23:05:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3873327104 bytes 2024-09-14T23:05:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3885524992 bytes 2024-09-14T23:05:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3929556992 bytes 2024-09-14T23:05:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3995710464 bytes 2024-09-14T23:05:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 4071524352 bytes 2024-09-14T23:05:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4094085120 bytes 2024-09-14T23:05:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4127041536 bytes 2024-09-14T23:05:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4179663872 bytes 2024-09-14T23:05:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4208506880 bytes 2024-09-14T23:05:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4283753472 bytes 2024-09-14T23:05:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4298802176 bytes 2024-09-14T23:05:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4306345984 bytes 2024-09-14T23:05:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4330524672 bytes 2024-09-14T23:05:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4345629696 bytes 2024-09-14T23:05:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4392276992 bytes 2024-09-14T23:05:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4451174400 bytes 2024-09-14T23:05:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4545985536 bytes 2024-09-14T23:05:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4610612224 bytes 2024-09-14T23:05:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4685341696 bytes 2024-09-14T23:05:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4747382784 bytes 2024-09-14T23:05:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 4802478080 bytes 2024-09-14T23:05:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11000 IOs, 4811328512 bytes 2024-09-14T23:05:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11100 IOs, 4841484288 bytes 2024-09-14T23:05:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11200 IOs, 4882620416 bytes 2024-09-14T23:05:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11300 IOs, 4945114112 bytes 2024-09-14T23:05:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11400 IOs, 4976777216 bytes 2024-09-14T23:05:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:05:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:05:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_29 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:05:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-14T23:05:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:05:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_29': 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 138c614f0751 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.138c614f0751 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:30 2024 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:30 2024 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:30 2024 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:05:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:05:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: ad5acb85-55a5-4fcf-abcf-6bf6f2766d18 2024-09-14T23:05:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:05:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:05:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-14T23:05:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:05:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:05:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T23:05:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:05:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:05:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 47603712 bytes 2024-09-14T23:05:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 99477504 bytes 2024-09-14T23:05:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 106142720 bytes 2024-09-14T23:05:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 120110080 bytes 2024-09-14T23:05:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 127079424 bytes 2024-09-14T23:05:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 193572864 bytes 2024-09-14T23:05:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 261178368 bytes 2024-09-14T23:05:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 332319744 bytes 2024-09-14T23:05:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 380024832 bytes 2024-09-14T23:05:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 401470464 bytes 2024-09-14T23:05:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 442265600 bytes 2024-09-14T23:05:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 459840512 bytes 2024-09-14T23:05:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 514572288 bytes 2024-09-14T23:05:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 576557056 bytes 2024-09-14T23:05:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 621405184 bytes 2024-09-14T23:05:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 697909248 bytes 2024-09-14T23:05:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 784903168 bytes 2024-09-14T23:05:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 811300864 bytes 2024-09-14T23:05:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 835303424 bytes 2024-09-14T23:05:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 856096768 bytes 2024-09-14T23:05:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 914924544 bytes 2024-09-14T23:05:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 951141376 bytes 2024-09-14T23:05:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1034286080 bytes 2024-09-14T23:05:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1056326656 bytes 2024-09-14T23:05:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1103258624 bytes 2024-09-14T23:05:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1137909760 bytes 2024-09-14T23:05:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1216197632 bytes 2024-09-14T23:05:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1306476544 bytes 2024-09-14T23:05:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1319926784 bytes 2024-09-14T23:05:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1356509184 bytes 2024-09-14T23:05:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1394609152 bytes 2024-09-14T23:05:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1463464960 bytes 2024-09-14T23:05:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1499222016 bytes 2024-09-14T23:05:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1544939520 bytes 2024-09-14T23:05:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1582379008 bytes 2024-09-14T23:05:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1595033600 bytes 2024-09-14T23:05:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1648396288 bytes 2024-09-14T23:05:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1692552192 bytes 2024-09-14T23:05:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1752263680 bytes 2024-09-14T23:05:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1770677248 bytes 2024-09-14T23:05:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1832279040 bytes 2024-09-14T23:05:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1904374784 bytes 2024-09-14T23:05:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1904973824 bytes 2024-09-14T23:05:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1949780992 bytes 2024-09-14T23:05:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1994005504 bytes 2024-09-14T23:05:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2022889472 bytes 2024-09-14T23:05:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2065702912 bytes 2024-09-14T23:05:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2095670272 bytes 2024-09-14T23:05:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:05:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:05:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_30 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-14T23:05:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:05:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:05:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:05:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:05:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2024-09-14T23:05:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:05:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:05:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-14T23:05:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:05:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-14T23:05:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:05:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-14T23:05:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_30': 2024-09-14T23:05:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:05:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:05:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:05:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1394398d48da 2024-09-14T23:05:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1394398d48da 2024-09-14T23:05:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:37 2024 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:37 2024 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:37 2024 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 2dd3a48d-e1ef-4b2b-b6fb-173793741ecc 2024-09-14T23:05:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:05:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:05:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-14T23:05:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:05:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:05:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=10 2024-09-14T23:05:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:05:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:05:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 18938880 bytes 2024-09-14T23:05:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 63884288 bytes 2024-09-14T23:05:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 85215232 bytes 2024-09-14T23:05:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 137956352 bytes 2024-09-14T23:05:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 158631936 bytes 2024-09-14T23:05:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 173810688 bytes 2024-09-14T23:05:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 264174592 bytes 2024-09-14T23:05:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 313076736 bytes 2024-09-14T23:05:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 360390656 bytes 2024-09-14T23:05:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 379385856 bytes 2024-09-14T23:05:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 419231744 bytes 2024-09-14T23:05:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 419848192 bytes 2024-09-14T23:05:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 501540864 bytes 2024-09-14T23:05:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 544939008 bytes 2024-09-14T23:05:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 617527296 bytes 2024-09-14T23:05:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 638793728 bytes 2024-09-14T23:05:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 661280768 bytes 2024-09-14T23:05:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 727148544 bytes 2024-09-14T23:05:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 789989376 bytes 2024-09-14T23:05:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 802112512 bytes 2024-09-14T23:05:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 832448512 bytes 2024-09-14T23:05:41.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 860528640 bytes 2024-09-14T23:05:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 887138304 bytes 2024-09-14T23:05:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 942168064 bytes 2024-09-14T23:05:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1025135616 bytes 2024-09-14T23:05:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1076032512 bytes 2024-09-14T23:05:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1113074688 bytes 2024-09-14T23:05:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1118804992 bytes 2024-09-14T23:05:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1144500224 bytes 2024-09-14T23:05:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1148102656 bytes 2024-09-14T23:05:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1148738560 bytes 2024-09-14T23:05:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1173832704 bytes 2024-09-14T23:05:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1200773120 bytes 2024-09-14T23:05:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1246430208 bytes 2024-09-14T23:05:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1285745664 bytes 2024-09-14T23:05:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1337104384 bytes 2024-09-14T23:05:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1358884864 bytes 2024-09-14T23:05:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1359482880 bytes 2024-09-14T23:05:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1396038656 bytes 2024-09-14T23:05:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1440864256 bytes 2024-09-14T23:05:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1491118080 bytes 2024-09-14T23:05:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1617697792 bytes 2024-09-14T23:05:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1696447488 bytes 2024-09-14T23:05:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1763212288 bytes 2024-09-14T23:05:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1773914112 bytes 2024-09-14T23:05:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1795740672 bytes 2024-09-14T23:05:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1800546304 bytes 2024-09-14T23:05:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1835125760 bytes 2024-09-14T23:05:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1851276288 bytes 2024-09-14T23:05:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1894841344 bytes 2024-09-14T23:05:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 1959493632 bytes 2024-09-14T23:05:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 1995288576 bytes 2024-09-14T23:05:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2021098496 bytes 2024-09-14T23:05:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2065695744 bytes 2024-09-14T23:05:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2105280512 bytes 2024-09-14T23:05:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2142737408 bytes 2024-09-14T23:05:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2184590336 bytes 2024-09-14T23:05:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2209042432 bytes 2024-09-14T23:05:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2267749376 bytes 2024-09-14T23:05:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2307827712 bytes 2024-09-14T23:05:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2308472832 bytes 2024-09-14T23:05:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2350081024 bytes 2024-09-14T23:05:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2419223552 bytes 2024-09-14T23:05:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2465462272 bytes 2024-09-14T23:05:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2494162944 bytes 2024-09-14T23:05:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2533165056 bytes 2024-09-14T23:05:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2575915008 bytes 2024-09-14T23:05:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2608104448 bytes 2024-09-14T23:05:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2649280512 bytes 2024-09-14T23:05:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2702729216 bytes 2024-09-14T23:05:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2752196608 bytes 2024-09-14T23:05:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2788361216 bytes 2024-09-14T23:05:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2803781632 bytes 2024-09-14T23:05:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2855714816 bytes 2024-09-14T23:05:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 2919335936 bytes 2024-09-14T23:05:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 2953979904 bytes 2024-09-14T23:05:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 2971655168 bytes 2024-09-14T23:05:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3004286976 bytes 2024-09-14T23:05:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3084039168 bytes 2024-09-14T23:05:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3149751296 bytes 2024-09-14T23:05:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3206013952 bytes 2024-09-14T23:05:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3212819456 bytes 2024-09-14T23:05:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3213477888 bytes 2024-09-14T23:05:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:05:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:05:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_31 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-14T23:05:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:05:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:05:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:05:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:05:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-14T23:05:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:05:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-14T23:05:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_31': 2024-09-14T23:05:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 139ccb057662 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.139ccb057662 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:48 2024 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:48 2024 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:48 2024 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:05:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:05:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: d5f8f61a-4276-4546-8e0b-db256cf1d59a 2024-09-14T23:05:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:05:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:05:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-14T23:05:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:05:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:05:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=9 2024-09-14T23:05:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:05:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:05:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 104389632 bytes 2024-09-14T23:05:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 133609472 bytes 2024-09-14T23:05:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 152905728 bytes 2024-09-14T23:05:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 197134336 bytes 2024-09-14T23:05:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 239367168 bytes 2024-09-14T23:05:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 267624448 bytes 2024-09-14T23:05:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 369561600 bytes 2024-09-14T23:05:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 411919360 bytes 2024-09-14T23:05:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 460059648 bytes 2024-09-14T23:05:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 510015488 bytes 2024-09-14T23:05:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 563290112 bytes 2024-09-14T23:05:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 606060544 bytes 2024-09-14T23:05:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 675396608 bytes 2024-09-14T23:05:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 680279040 bytes 2024-09-14T23:05:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 694047744 bytes 2024-09-14T23:05:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 811453440 bytes 2024-09-14T23:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 818291712 bytes 2024-09-14T23:05:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 859393024 bytes 2024-09-14T23:05:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 916356096 bytes 2024-09-14T23:05:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 1019878400 bytes 2024-09-14T23:05:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 1072650240 bytes 2024-09-14T23:05:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1137077248 bytes 2024-09-14T23:05:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1189682176 bytes 2024-09-14T23:05:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1214355456 bytes 2024-09-14T23:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1214982144 bytes 2024-09-14T23:05:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1229632512 bytes 2024-09-14T23:05:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1286900736 bytes 2024-09-14T23:05:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1346527232 bytes 2024-09-14T23:05:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1357693952 bytes 2024-09-14T23:05:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1397842944 bytes 2024-09-14T23:05:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1441029120 bytes 2024-09-14T23:05:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1461400576 bytes 2024-09-14T23:05:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1525199872 bytes 2024-09-14T23:05:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1556222976 bytes 2024-09-14T23:05:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1556827136 bytes 2024-09-14T23:05:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1590462464 bytes 2024-09-14T23:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1683225600 bytes 2024-09-14T23:05:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1744302080 bytes 2024-09-14T23:05:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1767230464 bytes 2024-09-14T23:05:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1821590528 bytes 2024-09-14T23:05:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1913861120 bytes 2024-09-14T23:05:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1943590912 bytes 2024-09-14T23:05:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1965147136 bytes 2024-09-14T23:05:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 2002657280 bytes 2024-09-14T23:05:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2069271552 bytes 2024-09-14T23:05:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2121211904 bytes 2024-09-14T23:05:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2150590464 bytes 2024-09-14T23:05:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2187435008 bytes 2024-09-14T23:05:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2256271360 bytes 2024-09-14T23:05:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2287240192 bytes 2024-09-14T23:05:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2320292864 bytes 2024-09-14T23:05:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2411784192 bytes 2024-09-14T23:05:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2468771840 bytes 2024-09-14T23:05:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2496898048 bytes 2024-09-14T23:05:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2541189120 bytes 2024-09-14T23:05:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2600232960 bytes 2024-09-14T23:05:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2655693824 bytes 2024-09-14T23:05:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2707094528 bytes 2024-09-14T23:05:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2733282304 bytes 2024-09-14T23:05:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2845142016 bytes 2024-09-14T23:05:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2869000192 bytes 2024-09-14T23:05:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2911096832 bytes 2024-09-14T23:05:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2979297280 bytes 2024-09-14T23:05:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 3027904512 bytes 2024-09-14T23:05:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 3083948032 bytes 2024-09-14T23:05:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 3135481856 bytes 2024-09-14T23:05:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 3174937600 bytes 2024-09-14T23:05:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 3178271744 bytes 2024-09-14T23:05:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3203918848 bytes 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_32 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:05:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:05:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:05:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-14T23:05:59.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:05:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-14T23:05:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_32': 2024-09-14T23:05:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:05:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:05:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:05:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13a4e6a1aef5 2024-09-14T23:05:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13a4e6a1aef5 2024-09-14T23:05:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:05:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:05:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:05:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:05:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:58 2024 2024-09-14T23:05:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:58 2024 2024-09-14T23:05:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:58 2024 2024-09-14T23:05:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:05:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:05:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 79596924-7c99-4578-ab4b-9b96094cd7b2 2024-09-14T23:05:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:05:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:05:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-14T23:05:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:05:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:05:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:05:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:05:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=7 2024-09-14T23:05:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:05:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:05:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 74821632 bytes 2024-09-14T23:05:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 99661824 bytes 2024-09-14T23:05:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 134305792 bytes 2024-09-14T23:05:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 170989568 bytes 2024-09-14T23:05:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 188735488 bytes 2024-09-14T23:05:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 203778048 bytes 2024-09-14T23:06:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 219913216 bytes 2024-09-14T23:06:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 267414528 bytes 2024-09-14T23:06:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 306585600 bytes 2024-09-14T23:06:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 330126336 bytes 2024-09-14T23:06:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 377657344 bytes 2024-09-14T23:06:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 413953024 bytes 2024-09-14T23:06:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 486813696 bytes 2024-09-14T23:06:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 510702592 bytes 2024-09-14T23:06:00.912 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9026397911776547) > weight (0.25) 2024-09-14T23:06:00.912 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5315227922967228) > weight (0.25) 2024-09-14T23:06:00.912 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.747251247464063) > weight (0.25) 2024-09-14T23:06:00.912 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-14T23:06:00.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-14T23:06:00.913 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-14T23:06:00.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:06:00.912+0000 7f03437c6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 41448) UID: 1000 2024-09-14T23:06:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 541679616 bytes 2024-09-14T23:06:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 569249792 bytes 2024-09-14T23:06:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 618667008 bytes 2024-09-14T23:06:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 672283648 bytes 2024-09-14T23:06:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 715791360 bytes 2024-09-14T23:06:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 738114560 bytes 2024-09-14T23:06:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 782757888 bytes 2024-09-14T23:06:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 824053760 bytes 2024-09-14T23:06:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 845260800 bytes 2024-09-14T23:06:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 883593216 bytes 2024-09-14T23:06:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 956797952 bytes 2024-09-14T23:06:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1010758656 bytes 2024-09-14T23:06:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1023132672 bytes 2024-09-14T23:06:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1060851712 bytes 2024-09-14T23:06:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1135910912 bytes 2024-09-14T23:06:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1201395712 bytes 2024-09-14T23:06:02.914 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-14T23:06:02.914 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:06:02.914 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-14T23:06:02.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-14T23:06:02.915 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-14T23:06:02.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-14T23:06:02.918 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-14T23:06:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1217680384 bytes 2024-09-14T23:06:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1301012480 bytes 2024-09-14T23:06:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1355589632 bytes 2024-09-14T23:06:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1395108864 bytes 2024-09-14T23:06:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1427406848 bytes 2024-09-14T23:06:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1487347712 bytes 2024-09-14T23:06:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1547238400 bytes 2024-09-14T23:06:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1590495232 bytes 2024-09-14T23:06:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1690137600 bytes 2024-09-14T23:06:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1690736640 bytes 2024-09-14T23:06:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1720309760 bytes 2024-09-14T23:06:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1766653952 bytes 2024-09-14T23:06:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1805640704 bytes 2024-09-14T23:06:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1806243840 bytes 2024-09-14T23:06:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1806875648 bytes 2024-09-14T23:06:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1898329088 bytes 2024-09-14T23:06:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1917795328 bytes 2024-09-14T23:06:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1984134144 bytes 2024-09-14T23:06:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2008432640 bytes 2024-09-14T23:06:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2040534016 bytes 2024-09-14T23:06:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2086916096 bytes 2024-09-14T23:06:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2087495680 bytes 2024-09-14T23:06:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2135865344 bytes 2024-09-14T23:06:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2170681344 bytes 2024-09-14T23:06:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2209130496 bytes 2024-09-14T23:06:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2245951488 bytes 2024-09-14T23:06:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2286264320 bytes 2024-09-14T23:06:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:06:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:06:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_33 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:06:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-14T23:06:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:06:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:06:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:06:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:06:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2024-09-14T23:06:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:06:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:06:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-14T23:06:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:06:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-14T23:06:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:06:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_33': 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13b68c0eb22 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13b68c0eb22 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:06:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:06 2024 2024-09-14T23:06:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:06 2024 2024-09-14T23:06:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:06 2024 2024-09-14T23:06:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:06:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:06:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 4c528d36-e484-45fe-9609-82252ec0ca3f 2024-09-14T23:06:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:06:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:06:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-14T23:06:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:06:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:06:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=13 2024-09-14T23:06:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:06:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:06:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 29489152 bytes 2024-09-14T23:06:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 46286848 bytes 2024-09-14T23:06:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 105003008 bytes 2024-09-14T23:06:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 166874112 bytes 2024-09-14T23:06:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 238081024 bytes 2024-09-14T23:06:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 252286976 bytes 2024-09-14T23:06:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 276610048 bytes 2024-09-14T23:06:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 297350144 bytes 2024-09-14T23:06:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 306272256 bytes 2024-09-14T23:06:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 333523968 bytes 2024-09-14T23:06:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 336790528 bytes 2024-09-14T23:06:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 379385856 bytes 2024-09-14T23:06:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 380015616 bytes 2024-09-14T23:06:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 423891968 bytes 2024-09-14T23:06:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 476236800 bytes 2024-09-14T23:06:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 509782016 bytes 2024-09-14T23:06:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 535287808 bytes 2024-09-14T23:06:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 572014592 bytes 2024-09-14T23:06:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 681123840 bytes 2024-09-14T23:06:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 760054784 bytes 2024-09-14T23:06:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 818025472 bytes 2024-09-14T23:06:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 839791616 bytes 2024-09-14T23:06:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 867892224 bytes 2024-09-14T23:06:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 915228672 bytes 2024-09-14T23:06:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 931976192 bytes 2024-09-14T23:06:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 963378176 bytes 2024-09-14T23:06:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1023618048 bytes 2024-09-14T23:06:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1084279808 bytes 2024-09-14T23:06:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1122260992 bytes 2024-09-14T23:06:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1155991552 bytes 2024-09-14T23:06:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1246647296 bytes 2024-09-14T23:06:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1273255936 bytes 2024-09-14T23:06:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1293520896 bytes 2024-09-14T23:06:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1310179328 bytes 2024-09-14T23:06:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1356164096 bytes 2024-09-14T23:06:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1451546624 bytes 2024-09-14T23:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1512630272 bytes 2024-09-14T23:06:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1556759552 bytes 2024-09-14T23:06:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1607206912 bytes 2024-09-14T23:06:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1666709504 bytes 2024-09-14T23:06:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1724679168 bytes 2024-09-14T23:06:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1794112512 bytes 2024-09-14T23:06:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1817243648 bytes 2024-09-14T23:06:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1854809088 bytes 2024-09-14T23:06:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1946089472 bytes 2024-09-14T23:06:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1983382528 bytes 2024-09-14T23:06:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2064787456 bytes 2024-09-14T23:06:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2094110720 bytes 2024-09-14T23:06:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2115225600 bytes 2024-09-14T23:06:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2230537216 bytes 2024-09-14T23:06:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2249641984 bytes 2024-09-14T23:06:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2290890752 bytes 2024-09-14T23:06:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2343170048 bytes 2024-09-14T23:06:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2343819264 bytes 2024-09-14T23:06:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2390811648 bytes 2024-09-14T23:06:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2420101120 bytes 2024-09-14T23:06:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2466379776 bytes 2024-09-14T23:06:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2517243904 bytes 2024-09-14T23:06:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2570149888 bytes 2024-09-14T23:06:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2570763264 bytes 2024-09-14T23:06:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2595513344 bytes 2024-09-14T23:06:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2627052544 bytes 2024-09-14T23:06:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2672822272 bytes 2024-09-14T23:06:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2706577408 bytes 2024-09-14T23:06:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2711744512 bytes 2024-09-14T23:06:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2712376320 bytes 2024-09-14T23:06:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2792193024 bytes 2024-09-14T23:06:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2801441792 bytes 2024-09-14T23:06:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2824850432 bytes 2024-09-14T23:06:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2875974656 bytes 2024-09-14T23:06:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2919091200 bytes 2024-09-14T23:06:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2951021568 bytes 2024-09-14T23:06:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3012021248 bytes 2024-09-14T23:06:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3070958592 bytes 2024-09-14T23:06:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3147955200 bytes 2024-09-14T23:06:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3156653056 bytes 2024-09-14T23:06:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3157268480 bytes 2024-09-14T23:06:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3238849536 bytes 2024-09-14T23:06:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3302275072 bytes 2024-09-14T23:06:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3320816640 bytes 2024-09-14T23:06:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3357647872 bytes 2024-09-14T23:06:17.016 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8633029065118598) > weight (0.25) 2024-09-14T23:06:17.016 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.449286137406556) > weight (0.25) 2024-09-14T23:06:17.016 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7814464262859221) > weight (0.25) 2024-09-14T23:06:17.016 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-14T23:06:17.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-14T23:06:17.017 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-14T23:06:17.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:06:17.016+0000 7fcd6dcb5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 43307) UID: 1000 2024-09-14T23:06:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3389476864 bytes 2024-09-14T23:06:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3454962688 bytes 2024-09-14T23:06:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3546819584 bytes 2024-09-14T23:06:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3566236672 bytes 2024-09-14T23:06:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3604513792 bytes 2024-09-14T23:06:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3620567040 bytes 2024-09-14T23:06:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3661874176 bytes 2024-09-14T23:06:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3752752128 bytes 2024-09-14T23:06:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3787265024 bytes 2024-09-14T23:06:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:06:18.182+0000 7fdfb75e9640 -1 rbd::mirror::ImageReplayer: 0x55ad5b0da780 [3/1183b733-6e8d-4fa0-bcc5-8cdba88bdd37] start: already running 2024-09-14T23:06:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:06:18.182+0000 7fdfc4603640 -1 rbd::mirror::ImageReplayer: 0x55ad5b0db400 [3/5b5b353a-9976-4517-9150-396dd4235407] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:06:18.182+0000 7fdfc3e02640 -1 rbd::mirror::ImageReplayer: 0x55ad5b0db400 [3/5b5b353a-9976-4517-9150-396dd4235407] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:06:18.182+0000 7fdfb75e9640 -1 rbd::mirror::ImageReplayer: 0x55ad5b0db400 [3/5b5b353a-9976-4517-9150-396dd4235407] start: already running 2024-09-14T23:06:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:06:18.182+0000 7fdfc4603640 -1 rbd::mirror::ImageReplayer: 0x55ad5b0da280 [3/d53f0a91-12f4-4ba8-846e-d09778cdee6d] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:06:18.182+0000 7fdfc3e02640 -1 rbd::mirror::ImageReplayer: 0x55ad5b0da280 [3/d53f0a91-12f4-4ba8-846e-d09778cdee6d] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:06:18.182+0000 7fdfc4603640 -1 rbd::mirror::ImageReplayer: 0x55ad5b0db900 [3/a848ce88-5df8-4b90-b6a1-72b3261b72bc] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:06:18.182+0000 7fdfc3e02640 -1 rbd::mirror::ImageReplayer: 0x55ad5b0db900 [3/a848ce88-5df8-4b90-b6a1-72b3261b72bc] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:06:18.186+0000 7f1679df8640 -1 rbd::mirror::ImageReplayer: 0x55e358da6780 [3/63095e72-2322-434d-9a80-ed8a5dcddbff] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:06:18.186+0000 7f16795f7640 -1 rbd::mirror::ImageReplayer: 0x55e358da6780 [3/63095e72-2322-434d-9a80-ed8a5dcddbff] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:06:18.186+0000 7f1679df8640 -1 rbd::mirror::ImageReplayer: 0x55e358480280 [3/13ef0e31-a893-45b8-a2c7-d21116cdc208] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:06:18.186+0000 7f16795f7640 -1 rbd::mirror::ImageReplayer: 0x55e358480280 [3/13ef0e31-a893-45b8-a2c7-d21116cdc208] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:06:18.186+0000 7f166cdde640 -1 rbd::mirror::ImageReplayer: 0x55e358480280 [3/13ef0e31-a893-45b8-a2c7-d21116cdc208] start: already running 2024-09-14T23:06:18.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:06:18.186+0000 7f1679df8640 -1 rbd::mirror::ImageReplayer: 0x55e358da7b80 [3/ad5acb85-55a5-4fcf-abcf-6bf6f2766d18] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:06:18.186+0000 7f16795f7640 -1 rbd::mirror::ImageReplayer: 0x55e358da7b80 [3/ad5acb85-55a5-4fcf-abcf-6bf6f2766d18] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:06:18.197+0000 7f8625955640 -1 rbd::mirror::ImageReplayer: 0x5647f5973180 [3/4c9c9805-ba1a-48da-a08b-8de70de557b3] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:06:18.198+0000 7f8626156640 -1 rbd::mirror::ImageReplayer: 0x5647f5973180 [3/4c9c9805-ba1a-48da-a08b-8de70de557b3] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:06:18.198+0000 7f8625955640 -1 rbd::mirror::ImageReplayer: 0x5647f5973400 [3/d02fb2d0-2bf7-49a8-86d4-ecd3c3240c86] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:06:18.198+0000 7f8626156640 -1 rbd::mirror::ImageReplayer: 0x5647f5973400 [3/d02fb2d0-2bf7-49a8-86d4-ecd3c3240c86] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:06:18.198+0000 7f8625955640 -1 rbd::mirror::ImageReplayer: 0x5647f5972280 [3/45c698d2-2f4b-4209-84b4-d26cd2e23680] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:06:18.198+0000 7f8626156640 -1 rbd::mirror::ImageReplayer: 0x5647f5972280 [3/45c698d2-2f4b-4209-84b4-d26cd2e23680] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:06:18.198+0000 7f8625955640 -1 rbd::mirror::ImageReplayer: 0x5647f5972a00 [3/ff85a14f-4301-4076-86de-a3f6e6b1ddd9] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:06:18.198+0000 7f8626156640 -1 rbd::mirror::ImageReplayer: 0x5647f5972a00 [3/ff85a14f-4301-4076-86de-a3f6e6b1ddd9] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:06:18.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:06:18.198+0000 7f166c5dd640 -1 rbd::mirror::ImageReplayer: 0x55e358da7400 [3/16ceb1c6-1a5c-4b7b-8b20-da6200f477b6] start: already running 2024-09-14T23:06:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3826654208 bytes 2024-09-14T23:06:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3891025920 bytes 2024-09-14T23:06:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3971301376 bytes 2024-09-14T23:06:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4079464448 bytes 2024-09-14T23:06:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4123378688 bytes 2024-09-14T23:06:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4174154752 bytes 2024-09-14T23:06:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4206089216 bytes 2024-09-14T23:06:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4255892480 bytes 2024-09-14T23:06:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4257837056 bytes 2024-09-14T23:06:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4312306688 bytes 2024-09-14T23:06:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4321241088 bytes 2024-09-14T23:06:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4336053248 bytes 2024-09-14T23:06:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4355709952 bytes 2024-09-14T23:06:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4420705280 bytes 2024-09-14T23:06:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4472371200 bytes 2024-09-14T23:06:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4520300544 bytes 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_34 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:06:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2024-09-14T23:06:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:06:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:06:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-14T23:06:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:06:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-14T23:06:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:06:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_34': 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13be5b24fb2f 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13be5b24fb2f 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:20 2024 2024-09-14T23:06:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:20 2024 2024-09-14T23:06:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:20 2024 2024-09-14T23:06:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:06:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:06:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: b3b49072-90fc-4a20-87f1-ba0c9f24cea9 2024-09-14T23:06:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:06:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:06:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-14T23:06:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:06:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=7 2024-09-14T23:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:06:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 3428352 bytes 2024-09-14T23:06:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 36326400 bytes 2024-09-14T23:06:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 72736768 bytes 2024-09-14T23:06:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 101209088 bytes 2024-09-14T23:06:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 184597504 bytes 2024-09-14T23:06:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 279284736 bytes 2024-09-14T23:06:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 279892992 bytes 2024-09-14T23:06:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 300447744 bytes 2024-09-14T23:06:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 354840576 bytes 2024-09-14T23:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 429598720 bytes 2024-09-14T23:06:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 481784832 bytes 2024-09-14T23:06:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 537322496 bytes 2024-09-14T23:06:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 584719360 bytes 2024-09-14T23:06:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 623197184 bytes 2024-09-14T23:06:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 715368448 bytes 2024-09-14T23:06:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 778140672 bytes 2024-09-14T23:06:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 862969856 bytes 2024-09-14T23:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 867782656 bytes 2024-09-14T23:06:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 891182080 bytes 2024-09-14T23:06:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 903956480 bytes 2024-09-14T23:06:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 945526784 bytes 2024-09-14T23:06:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1004127232 bytes 2024-09-14T23:06:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1008214016 bytes 2024-09-14T23:06:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1019552768 bytes 2024-09-14T23:06:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1081771008 bytes 2024-09-14T23:06:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1102216192 bytes 2024-09-14T23:06:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1129127936 bytes 2024-09-14T23:06:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1135946752 bytes 2024-09-14T23:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1181505536 bytes 2024-09-14T23:06:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1228908544 bytes 2024-09-14T23:06:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1279009792 bytes 2024-09-14T23:06:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1297707008 bytes 2024-09-14T23:06:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1316530176 bytes 2024-09-14T23:06:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1366530048 bytes 2024-09-14T23:06:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1393820672 bytes 2024-09-14T23:06:25.017 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-14T23:06:25.017 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:06:25.018 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-14T23:06:25.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T23:06:25.018 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T23:06:25.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T23:06:25.022 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-09-14T23:06:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1479313408 bytes 2024-09-14T23:06:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1531789312 bytes 2024-09-14T23:06:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1582688256 bytes 2024-09-14T23:06:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1651744768 bytes 2024-09-14T23:06:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1654344704 bytes 2024-09-14T23:06:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1747712000 bytes 2024-09-14T23:06:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1770149888 bytes 2024-09-14T23:06:25.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1787110400 bytes 2024-09-14T23:06:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1820328960 bytes 2024-09-14T23:06:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1938438144 bytes 2024-09-14T23:06:26.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1939037184 bytes 2024-09-14T23:06:26.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1958760448 bytes 2024-09-14T23:06:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1989646336 bytes 2024-09-14T23:06:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1993993216 bytes 2024-09-14T23:06:26.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2049812480 bytes 2024-09-14T23:06:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2131495936 bytes 2024-09-14T23:06:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2181615616 bytes 2024-09-14T23:06:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2187114496 bytes 2024-09-14T23:06:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2244097024 bytes 2024-09-14T23:06:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2310699008 bytes 2024-09-14T23:06:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2321150976 bytes 2024-09-14T23:06:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:06:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:06:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:06:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:06:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_35 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-14T23:06:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:06:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:06:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:06:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:06:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2024-09-14T23:06:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:06:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:06:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-14T23:06:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:06:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-14T23:06:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:06:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-14T23:06:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_35': 2024-09-14T23:06:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13cd693f3956 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13cd693f3956 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:27 2024 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:27 2024 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:27 2024 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 62edd50a-04e9-49e3-95e0-0652ce1bea79 2024-09-14T23:06:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:06:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:06:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-14T23:06:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:06:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:06:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T23:06:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:06:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:06:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 42521600 bytes 2024-09-14T23:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 153300992 bytes 2024-09-14T23:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 215012352 bytes 2024-09-14T23:06:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 260787200 bytes 2024-09-14T23:06:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 346732544 bytes 2024-09-14T23:06:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 411910144 bytes 2024-09-14T23:06:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 430082048 bytes 2024-09-14T23:06:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 459744256 bytes 2024-09-14T23:06:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 507945984 bytes 2024-09-14T23:06:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 529909760 bytes 2024-09-14T23:06:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 574942208 bytes 2024-09-14T23:06:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 617942016 bytes 2024-09-14T23:06:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 660899840 bytes 2024-09-14T23:06:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 707809280 bytes 2024-09-14T23:06:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 735007744 bytes 2024-09-14T23:06:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 763887616 bytes 2024-09-14T23:06:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 783943680 bytes 2024-09-14T23:06:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 852052992 bytes 2024-09-14T23:06:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 871424000 bytes 2024-09-14T23:06:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 939564032 bytes 2024-09-14T23:06:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 981480448 bytes 2024-09-14T23:06:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1025827840 bytes 2024-09-14T23:06:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1069413376 bytes 2024-09-14T23:06:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1080638464 bytes 2024-09-14T23:06:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1109921792 bytes 2024-09-14T23:06:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1153750016 bytes 2024-09-14T23:06:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1209627648 bytes 2024-09-14T23:06:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1235728384 bytes 2024-09-14T23:06:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1281833984 bytes 2024-09-14T23:06:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1328815104 bytes 2024-09-14T23:06:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1387044864 bytes 2024-09-14T23:06:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1444256768 bytes 2024-09-14T23:06:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1456315392 bytes 2024-09-14T23:06:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1470842880 bytes 2024-09-14T23:06:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1516647424 bytes 2024-09-14T23:06:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1578334208 bytes 2024-09-14T23:06:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1625737216 bytes 2024-09-14T23:06:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1654569984 bytes 2024-09-14T23:06:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1701149696 bytes 2024-09-14T23:06:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1731401728 bytes 2024-09-14T23:06:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1762241536 bytes 2024-09-14T23:06:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1775025152 bytes 2024-09-14T23:06:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1861181440 bytes 2024-09-14T23:06:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1891489792 bytes 2024-09-14T23:06:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1931052032 bytes 2024-09-14T23:06:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2010699776 bytes 2024-09-14T23:06:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2045118464 bytes 2024-09-14T23:06:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2096223232 bytes 2024-09-14T23:06:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_36 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:06:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2024-09-14T23:06:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:06:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:06:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-14T23:06:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:06:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-14T23:06:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:06:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-14T23:06:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_36': 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13d53e9f79d2 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13d53e9f79d2 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:34 2024 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:34 2024 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:34 2024 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:06:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:06:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 9def2b16-9e96-4962-a4ee-3a7a75bafdd6 2024-09-14T23:06:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:06:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:06:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-14T23:06:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:06:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:06:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=10 2024-09-14T23:06:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:06:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:06:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 62290944 bytes 2024-09-14T23:06:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 109549568 bytes 2024-09-14T23:06:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 147138560 bytes 2024-09-14T23:06:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 175381504 bytes 2024-09-14T23:06:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 221110272 bytes 2024-09-14T23:06:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 230690816 bytes 2024-09-14T23:06:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 262693888 bytes 2024-09-14T23:06:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 308329472 bytes 2024-09-14T23:06:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 371047424 bytes 2024-09-14T23:06:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 397996032 bytes 2024-09-14T23:06:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 467608576 bytes 2024-09-14T23:06:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 498955264 bytes 2024-09-14T23:06:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 545812480 bytes 2024-09-14T23:06:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 603915264 bytes 2024-09-14T23:06:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 644546560 bytes 2024-09-14T23:06:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 750167040 bytes 2024-09-14T23:06:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 779780096 bytes 2024-09-14T23:06:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 872473600 bytes 2024-09-14T23:06:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 873066496 bytes 2024-09-14T23:06:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 891297792 bytes 2024-09-14T23:06:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 908303360 bytes 2024-09-14T23:06:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 915676160 bytes 2024-09-14T23:06:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 956286976 bytes 2024-09-14T23:06:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 989054976 bytes 2024-09-14T23:06:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1029090304 bytes 2024-09-14T23:06:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1128658944 bytes 2024-09-14T23:06:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1170868224 bytes 2024-09-14T23:06:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1210728448 bytes 2024-09-14T23:06:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1230903296 bytes 2024-09-14T23:06:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1319371776 bytes 2024-09-14T23:06:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1328890880 bytes 2024-09-14T23:06:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1385458688 bytes 2024-09-14T23:06:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1474829312 bytes 2024-09-14T23:06:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1511227392 bytes 2024-09-14T23:06:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1520858112 bytes 2024-09-14T23:06:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1621295104 bytes 2024-09-14T23:06:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1669692416 bytes 2024-09-14T23:06:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1688415232 bytes 2024-09-14T23:06:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1715804160 bytes 2024-09-14T23:06:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1784182784 bytes 2024-09-14T23:06:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1826099200 bytes 2024-09-14T23:06:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1866648576 bytes 2024-09-14T23:06:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1883959296 bytes 2024-09-14T23:06:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1912705024 bytes 2024-09-14T23:06:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1925848064 bytes 2024-09-14T23:06:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1965454336 bytes 2024-09-14T23:06:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2005679104 bytes 2024-09-14T23:06:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2029301760 bytes 2024-09-14T23:06:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2075272192 bytes 2024-09-14T23:06:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2087173120 bytes 2024-09-14T23:06:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2124518400 bytes 2024-09-14T23:06:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2172441600 bytes 2024-09-14T23:06:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2270490624 bytes 2024-09-14T23:06:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2304932864 bytes 2024-09-14T23:06:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2324086784 bytes 2024-09-14T23:06:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2365594624 bytes 2024-09-14T23:06:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2399884288 bytes 2024-09-14T23:06:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2498677760 bytes 2024-09-14T23:06:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2579391488 bytes 2024-09-14T23:06:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2628976640 bytes 2024-09-14T23:06:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2640514048 bytes 2024-09-14T23:06:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2679025664 bytes 2024-09-14T23:06:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2719279104 bytes 2024-09-14T23:06:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2789378048 bytes 2024-09-14T23:06:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2868702208 bytes 2024-09-14T23:06:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2900469760 bytes 2024-09-14T23:06:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2976607232 bytes 2024-09-14T23:06:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 3003878400 bytes 2024-09-14T23:06:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3037686784 bytes 2024-09-14T23:06:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3066142720 bytes 2024-09-14T23:06:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3085429760 bytes 2024-09-14T23:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3120456704 bytes 2024-09-14T23:06:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3209897984 bytes 2024-09-14T23:06:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3251181568 bytes 2024-09-14T23:06:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3282728960 bytes 2024-09-14T23:06:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3398105088 bytes 2024-09-14T23:06:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3434797056 bytes 2024-09-14T23:06:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3445183488 bytes 2024-09-14T23:06:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3503197184 bytes 2024-09-14T23:06:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3522968576 bytes 2024-09-14T23:06:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3589068800 bytes 2024-09-14T23:06:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:06:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:06:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:06:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:06:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_37 2024-09-14T23:06:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-14T23:06:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-14T23:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:06:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2024-09-14T23:06:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:06:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:06:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-14T23:06:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:06:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-14T23:06:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:06:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-14T23:06:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_37': 2024-09-14T23:06:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:06:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:06:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:06:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13de5f1b438e 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13de5f1b438e 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:45 2024 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:45 2024 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:45 2024 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: efd468e5-5b78-4874-94e4-101d543ad42f 2024-09-14T23:06:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:06:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:06:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-14T23:06:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:06:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:06:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=13 2024-09-14T23:06:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:06:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:06:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 16769024 bytes 2024-09-14T23:06:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 102362112 bytes 2024-09-14T23:06:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 136133632 bytes 2024-09-14T23:06:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 183945216 bytes 2024-09-14T23:06:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 260011008 bytes 2024-09-14T23:06:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 289758208 bytes 2024-09-14T23:06:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 298155008 bytes 2024-09-14T23:06:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 382480384 bytes 2024-09-14T23:06:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 408829952 bytes 2024-09-14T23:06:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 422815744 bytes 2024-09-14T23:06:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 507389952 bytes 2024-09-14T23:06:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 533376000 bytes 2024-09-14T23:06:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 533976064 bytes 2024-09-14T23:06:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 599435264 bytes 2024-09-14T23:06:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 620635136 bytes 2024-09-14T23:06:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 645786624 bytes 2024-09-14T23:06:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 685985792 bytes 2024-09-14T23:06:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 714935296 bytes 2024-09-14T23:06:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 816747520 bytes 2024-09-14T23:06:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 855846912 bytes 2024-09-14T23:06:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 918961152 bytes 2024-09-14T23:06:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 981020672 bytes 2024-09-14T23:06:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1014114304 bytes 2024-09-14T23:06:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1062222848 bytes 2024-09-14T23:06:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1112428544 bytes 2024-09-14T23:06:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1130527744 bytes 2024-09-14T23:06:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1179899904 bytes 2024-09-14T23:06:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1202614272 bytes 2024-09-14T23:06:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1228871680 bytes 2024-09-14T23:06:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1261821952 bytes 2024-09-14T23:06:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1288526848 bytes 2024-09-14T23:06:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1333876736 bytes 2024-09-14T23:06:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1385760768 bytes 2024-09-14T23:06:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1436615680 bytes 2024-09-14T23:06:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1448612864 bytes 2024-09-14T23:06:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1508105216 bytes 2024-09-14T23:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1548557312 bytes 2024-09-14T23:06:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1619798016 bytes 2024-09-14T23:06:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1620440064 bytes 2024-09-14T23:06:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1667341312 bytes 2024-09-14T23:06:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1723505664 bytes 2024-09-14T23:06:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1739716608 bytes 2024-09-14T23:06:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1749154816 bytes 2024-09-14T23:06:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1799779328 bytes 2024-09-14T23:06:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1824849920 bytes 2024-09-14T23:06:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1840519168 bytes 2024-09-14T23:06:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1860489216 bytes 2024-09-14T23:06:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1879783424 bytes 2024-09-14T23:06:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1898831872 bytes 2024-09-14T23:06:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2012626944 bytes 2024-09-14T23:06:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2063970304 bytes 2024-09-14T23:06:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2085179392 bytes 2024-09-14T23:06:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2121103360 bytes 2024-09-14T23:06:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2121711616 bytes 2024-09-14T23:06:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2184126464 bytes 2024-09-14T23:06:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2223246336 bytes 2024-09-14T23:06:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2253779968 bytes 2024-09-14T23:06:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2339453952 bytes 2024-09-14T23:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2397934592 bytes 2024-09-14T23:06:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2407058432 bytes 2024-09-14T23:06:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2431164416 bytes 2024-09-14T23:06:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2450391040 bytes 2024-09-14T23:06:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2516799488 bytes 2024-09-14T23:06:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2517404672 bytes 2024-09-14T23:06:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2585972736 bytes 2024-09-14T23:06:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2618617856 bytes 2024-09-14T23:06:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2652060672 bytes 2024-09-14T23:06:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2697897984 bytes 2024-09-14T23:06:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2710611968 bytes 2024-09-14T23:06:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2765673472 bytes 2024-09-14T23:06:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2772786176 bytes 2024-09-14T23:06:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2799308800 bytes 2024-09-14T23:06:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2885962752 bytes 2024-09-14T23:06:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2917555200 bytes 2024-09-14T23:06:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 2948102144 bytes 2024-09-14T23:06:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 2980151296 bytes 2024-09-14T23:06:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3007451136 bytes 2024-09-14T23:06:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3049955328 bytes 2024-09-14T23:06:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3119456256 bytes 2024-09-14T23:06:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3173624832 bytes 2024-09-14T23:06:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3174245376 bytes 2024-09-14T23:06:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3187558400 bytes 2024-09-14T23:06:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3211736064 bytes 2024-09-14T23:06:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3229336576 bytes 2024-09-14T23:06:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3264379904 bytes 2024-09-14T23:06:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3319956480 bytes 2024-09-14T23:06:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3387846656 bytes 2024-09-14T23:06:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3412214784 bytes 2024-09-14T23:06:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3463675904 bytes 2024-09-14T23:06:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3469978624 bytes 2024-09-14T23:06:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3501304832 bytes 2024-09-14T23:06:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3534859264 bytes 2024-09-14T23:06:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3601547264 bytes 2024-09-14T23:06:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3670148096 bytes 2024-09-14T23:06:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 3730165760 bytes 2024-09-14T23:06:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 3812598784 bytes 2024-09-14T23:06:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 3851488256 bytes 2024-09-14T23:06:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 3961469952 bytes 2024-09-14T23:06:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 3983865856 bytes 2024-09-14T23:06:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4007588864 bytes 2024-09-14T23:06:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4073540608 bytes 2024-09-14T23:06:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4158073856 bytes 2024-09-14T23:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4222395392 bytes 2024-09-14T23:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4260195328 bytes 2024-09-14T23:06:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4307450880 bytes 2024-09-14T23:06:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:06:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:06:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:06:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:06:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_38 2024-09-14T23:06:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-14T23:06:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:06:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:06:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:06:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:06:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:06:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:06:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-14T23:07:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:07:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_38': 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13e71482af6c 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13e71482af6c 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:59 2024 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:59 2024 2024-09-14T23:07:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:59 2024 2024-09-14T23:07:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:07:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:07:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 3f0fd191-d8d6-479a-bc37-05f42708eaf7 2024-09-14T23:07:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:07:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:07:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-14T23:07:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:07:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:07:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=5 2024-09-14T23:07:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:07:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:07:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 16982016 bytes 2024-09-14T23:07:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 78897152 bytes 2024-09-14T23:07:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 103942144 bytes 2024-09-14T23:07:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 117547008 bytes 2024-09-14T23:07:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 204262400 bytes 2024-09-14T23:07:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 315483136 bytes 2024-09-14T23:07:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 354931712 bytes 2024-09-14T23:07:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 395151360 bytes 2024-09-14T23:07:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 458306560 bytes 2024-09-14T23:07:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 502394880 bytes 2024-09-14T23:07:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 619112448 bytes 2024-09-14T23:07:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 646010880 bytes 2024-09-14T23:07:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 738679808 bytes 2024-09-14T23:07:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 754511872 bytes 2024-09-14T23:07:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 821934080 bytes 2024-09-14T23:07:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 861609984 bytes 2024-09-14T23:07:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 883935232 bytes 2024-09-14T23:07:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 907061248 bytes 2024-09-14T23:07:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 922846208 bytes 2024-09-14T23:07:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 937885696 bytes 2024-09-14T23:07:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 992140288 bytes 2024-09-14T23:07:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1053450240 bytes 2024-09-14T23:07:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1067279360 bytes 2024-09-14T23:07:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1084990464 bytes 2024-09-14T23:07:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1125758976 bytes 2024-09-14T23:07:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1193681920 bytes 2024-09-14T23:07:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1226358784 bytes 2024-09-14T23:07:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1272270848 bytes 2024-09-14T23:07:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1360427008 bytes 2024-09-14T23:07:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1447218176 bytes 2024-09-14T23:07:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1447839744 bytes 2024-09-14T23:07:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1528872960 bytes 2024-09-14T23:07:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1543619584 bytes 2024-09-14T23:07:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1592066048 bytes 2024-09-14T23:07:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1616553984 bytes 2024-09-14T23:07:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1677862912 bytes 2024-09-14T23:07:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1742520320 bytes 2024-09-14T23:07:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1791276032 bytes 2024-09-14T23:07:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1886431232 bytes 2024-09-14T23:07:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:07:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:07:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:07:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:07:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_39 2024-09-14T23:07:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-14T23:07:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2024-09-14T23:07:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:07:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:07:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-14T23:07:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:07:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-14T23:07:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:07:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-14T23:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_39': 2024-09-14T23:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13f0f2d5a61 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13f0f2d5a61 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:05 2024 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:05 2024 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:05 2024 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 2c75e949-74f5-488e-aeb1-ce0ee7bef852 2024-09-14T23:07:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:07:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:07:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-14T23:07:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:07:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:07:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=12 2024-09-14T23:07:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:07:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:07:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 71829504 bytes 2024-09-14T23:07:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 122407936 bytes 2024-09-14T23:07:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 139544576 bytes 2024-09-14T23:07:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 154372096 bytes 2024-09-14T23:07:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 178104320 bytes 2024-09-14T23:07:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 237161472 bytes 2024-09-14T23:07:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 312816640 bytes 2024-09-14T23:07:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 313434112 bytes 2024-09-14T23:07:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 362964992 bytes 2024-09-14T23:07:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 457362432 bytes 2024-09-14T23:07:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 490254336 bytes 2024-09-14T23:07:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 515736576 bytes 2024-09-14T23:07:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 566955008 bytes 2024-09-14T23:07:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 590554112 bytes 2024-09-14T23:07:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 679120896 bytes 2024-09-14T23:07:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 690262016 bytes 2024-09-14T23:07:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 703712256 bytes 2024-09-14T23:07:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 767570944 bytes 2024-09-14T23:07:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 787311616 bytes 2024-09-14T23:07:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 787934208 bytes 2024-09-14T23:07:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 797161472 bytes 2024-09-14T23:07:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 838094848 bytes 2024-09-14T23:07:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 962424832 bytes 2024-09-14T23:07:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 977438720 bytes 2024-09-14T23:07:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1060409344 bytes 2024-09-14T23:07:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1107601408 bytes 2024-09-14T23:07:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1108224000 bytes 2024-09-14T23:07:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1154764800 bytes 2024-09-14T23:07:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1231657984 bytes 2024-09-14T23:07:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1248856064 bytes 2024-09-14T23:07:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1321066496 bytes 2024-09-14T23:07:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1377892352 bytes 2024-09-14T23:07:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1385986048 bytes 2024-09-14T23:07:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1397761024 bytes 2024-09-14T23:07:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1428251648 bytes 2024-09-14T23:07:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1449169920 bytes 2024-09-14T23:07:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1497583616 bytes 2024-09-14T23:07:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1510586368 bytes 2024-09-14T23:07:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1595585536 bytes 2024-09-14T23:07:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1629191168 bytes 2024-09-14T23:07:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1649227776 bytes 2024-09-14T23:07:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1687348224 bytes 2024-09-14T23:07:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1720057856 bytes 2024-09-14T23:07:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1745533952 bytes 2024-09-14T23:07:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1801265152 bytes 2024-09-14T23:07:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1846109184 bytes 2024-09-14T23:07:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1876503552 bytes 2024-09-14T23:07:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1927986176 bytes 2024-09-14T23:07:12.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1970045952 bytes 2024-09-14T23:07:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2041116672 bytes 2024-09-14T23:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2050881536 bytes 2024-09-14T23:07:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2094810112 bytes 2024-09-14T23:07:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2127025152 bytes 2024-09-14T23:07:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2137928704 bytes 2024-09-14T23:07:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2171623424 bytes 2024-09-14T23:07:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2172246016 bytes 2024-09-14T23:07:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2178681856 bytes 2024-09-14T23:07:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2225779712 bytes 2024-09-14T23:07:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2226400256 bytes 2024-09-14T23:07:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2274596864 bytes 2024-09-14T23:07:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2299922432 bytes 2024-09-14T23:07:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2347013120 bytes 2024-09-14T23:07:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2411962368 bytes 2024-09-14T23:07:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2412610560 bytes 2024-09-14T23:07:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2452914176 bytes 2024-09-14T23:07:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2466338816 bytes 2024-09-14T23:07:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2496557056 bytes 2024-09-14T23:07:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2578903040 bytes 2024-09-14T23:07:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2709291008 bytes 2024-09-14T23:07:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2758540288 bytes 2024-09-14T23:07:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2786087936 bytes 2024-09-14T23:07:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2820905984 bytes 2024-09-14T23:07:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2845118464 bytes 2024-09-14T23:07:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2909766656 bytes 2024-09-14T23:07:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 2945176576 bytes 2024-09-14T23:07:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 2975304704 bytes 2024-09-14T23:07:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3024465920 bytes 2024-09-14T23:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3147282432 bytes 2024-09-14T23:07:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3224366080 bytes 2024-09-14T23:07:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3296504832 bytes 2024-09-14T23:07:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3297144832 bytes 2024-09-14T23:07:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3309670400 bytes 2024-09-14T23:07:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3327152128 bytes 2024-09-14T23:07:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3345471488 bytes 2024-09-14T23:07:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3362600960 bytes 2024-09-14T23:07:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3379529728 bytes 2024-09-14T23:07:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3453406208 bytes 2024-09-14T23:07:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3503433728 bytes 2024-09-14T23:07:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3552891904 bytes 2024-09-14T23:07:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3583512576 bytes 2024-09-14T23:07:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3613102080 bytes 2024-09-14T23:07:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3694309376 bytes 2024-09-14T23:07:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3712208896 bytes 2024-09-14T23:07:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3782024192 bytes 2024-09-14T23:07:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 3796573184 bytes 2024-09-14T23:07:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 3859766272 bytes 2024-09-14T23:07:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 3882616832 bytes 2024-09-14T23:07:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 3899877376 bytes 2024-09-14T23:07:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 3944571904 bytes 2024-09-14T23:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 3976619008 bytes 2024-09-14T23:07:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4010701824 bytes 2024-09-14T23:07:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:07:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_40 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2024-09-14T23:07:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:07:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:07:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-14T23:07:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:07:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-14T23:07:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:07:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_40': 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 13f88a038dd8 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.13f88a038dd8 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:07:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:18 2024 2024-09-14T23:07:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:18 2024 2024-09-14T23:07:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:18 2024 2024-09-14T23:07:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:07:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:07:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 8837ff64-098e-4da2-a725-5197d788c4b5 2024-09-14T23:07:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:07:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:07:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-14T23:07:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:07:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:07:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=7 2024-09-14T23:07:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:07:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:07:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 17153024 bytes 2024-09-14T23:07:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 110156800 bytes 2024-09-14T23:07:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 118001664 bytes 2024-09-14T23:07:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 141679616 bytes 2024-09-14T23:07:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 217721856 bytes 2024-09-14T23:07:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 288314368 bytes 2024-09-14T23:07:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 329242624 bytes 2024-09-14T23:07:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 408212480 bytes 2024-09-14T23:07:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 424346624 bytes 2024-09-14T23:07:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 464671744 bytes 2024-09-14T23:07:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 482012160 bytes 2024-09-14T23:07:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 593244160 bytes 2024-09-14T23:07:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 637336576 bytes 2024-09-14T23:07:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 672791552 bytes 2024-09-14T23:07:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 716161024 bytes 2024-09-14T23:07:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 784463872 bytes 2024-09-14T23:07:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 845735936 bytes 2024-09-14T23:07:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 867944448 bytes 2024-09-14T23:07:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 943822848 bytes 2024-09-14T23:07:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 966048768 bytes 2024-09-14T23:07:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 1032697856 bytes 2024-09-14T23:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1073874944 bytes 2024-09-14T23:07:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1096857600 bytes 2024-09-14T23:07:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1118010368 bytes 2024-09-14T23:07:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1141352448 bytes 2024-09-14T23:07:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1147951104 bytes 2024-09-14T23:07:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1244460032 bytes 2024-09-14T23:07:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1255215104 bytes 2024-09-14T23:07:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1304720384 bytes 2024-09-14T23:07:23.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1342262272 bytes 2024-09-14T23:07:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1423111168 bytes 2024-09-14T23:07:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1465577472 bytes 2024-09-14T23:07:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1467518976 bytes 2024-09-14T23:07:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1473555456 bytes 2024-09-14T23:07:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1530307584 bytes 2024-09-14T23:07:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1555490816 bytes 2024-09-14T23:07:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1599642624 bytes 2024-09-14T23:07:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1605512192 bytes 2024-09-14T23:07:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1675084800 bytes 2024-09-14T23:07:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1713037312 bytes 2024-09-14T23:07:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1724812288 bytes 2024-09-14T23:07:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1792357376 bytes 2024-09-14T23:07:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1827979264 bytes 2024-09-14T23:07:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1861814272 bytes 2024-09-14T23:07:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1917276160 bytes 2024-09-14T23:07:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1932268544 bytes 2024-09-14T23:07:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1998763008 bytes 2024-09-14T23:07:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2005664768 bytes 2024-09-14T23:07:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2064572416 bytes 2024-09-14T23:07:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2122117120 bytes 2024-09-14T23:07:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2200378368 bytes 2024-09-14T23:07:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2256269312 bytes 2024-09-14T23:07:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2290320384 bytes 2024-09-14T23:07:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2293137408 bytes 2024-09-14T23:07:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2357207040 bytes 2024-09-14T23:07:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2388378624 bytes 2024-09-14T23:07:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:07:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:07:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:07:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:07:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_41 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-14T23:07:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:07:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:07:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:07:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:07:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2024-09-14T23:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-14T23:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-14T23:07:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:07:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-14T23:07:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_41': 2024-09-14T23:07:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 140025ac5e88 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.140025ac5e88 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:26 2024 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:26 2024 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:26 2024 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:07:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 56fdc800-4370-46f4-a73e-74ac75619223 2024-09-14T23:07:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:07:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:07:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-14T23:07:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:07:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:07:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=9 2024-09-14T23:07:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:07:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:07:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 32930816 bytes 2024-09-14T23:07:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 45557760 bytes 2024-09-14T23:07:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 86793216 bytes 2024-09-14T23:07:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 128105472 bytes 2024-09-14T23:07:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 190382080 bytes 2024-09-14T23:07:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 217064448 bytes 2024-09-14T23:07:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 273747968 bytes 2024-09-14T23:07:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 339300352 bytes 2024-09-14T23:07:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 387552256 bytes 2024-09-14T23:07:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 444496896 bytes 2024-09-14T23:07:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 540613632 bytes 2024-09-14T23:07:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 595452928 bytes 2024-09-14T23:07:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 622705664 bytes 2024-09-14T23:07:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 677141504 bytes 2024-09-14T23:07:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 702238720 bytes 2024-09-14T23:07:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 766468096 bytes 2024-09-14T23:07:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 792724480 bytes 2024-09-14T23:07:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 816199680 bytes 2024-09-14T23:07:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 859748352 bytes 2024-09-14T23:07:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 910294016 bytes 2024-09-14T23:07:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 934993920 bytes 2024-09-14T23:07:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 977876992 bytes 2024-09-14T23:07:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1003852800 bytes 2024-09-14T23:07:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1054764032 bytes 2024-09-14T23:07:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1124799488 bytes 2024-09-14T23:07:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1149698048 bytes 2024-09-14T23:07:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1197777920 bytes 2024-09-14T23:07:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1208238080 bytes 2024-09-14T23:07:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1220476928 bytes 2024-09-14T23:07:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1233918976 bytes 2024-09-14T23:07:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1234527232 bytes 2024-09-14T23:07:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1250396160 bytes 2024-09-14T23:07:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1254732800 bytes 2024-09-14T23:07:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1300720640 bytes 2024-09-14T23:07:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1331283968 bytes 2024-09-14T23:07:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1356398592 bytes 2024-09-14T23:07:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1405681664 bytes 2024-09-14T23:07:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1450763264 bytes 2024-09-14T23:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1496283136 bytes 2024-09-14T23:07:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1527730176 bytes 2024-09-14T23:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1598204928 bytes 2024-09-14T23:07:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1602482176 bytes 2024-09-14T23:07:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1680945152 bytes 2024-09-14T23:07:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1693756416 bytes 2024-09-14T23:07:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1724568576 bytes 2024-09-14T23:07:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1746973696 bytes 2024-09-14T23:07:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1747580928 bytes 2024-09-14T23:07:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1780660224 bytes 2024-09-14T23:07:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1830075392 bytes 2024-09-14T23:07:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1931709440 bytes 2024-09-14T23:07:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2010771456 bytes 2024-09-14T23:07:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2054363136 bytes 2024-09-14T23:07:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2116579328 bytes 2024-09-14T23:07:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2159926272 bytes 2024-09-14T23:07:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2210460672 bytes 2024-09-14T23:07:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2279732224 bytes 2024-09-14T23:07:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2317760512 bytes 2024-09-14T23:07:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2331874304 bytes 2024-09-14T23:07:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2342720512 bytes 2024-09-14T23:07:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2359289856 bytes 2024-09-14T23:07:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2372237312 bytes 2024-09-14T23:07:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2407557120 bytes 2024-09-14T23:07:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2446888960 bytes 2024-09-14T23:07:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2453743616 bytes 2024-09-14T23:07:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2530289664 bytes 2024-09-14T23:07:35.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2554552320 bytes 2024-09-14T23:07:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2594565120 bytes 2024-09-14T23:07:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2627497984 bytes 2024-09-14T23:07:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2715708416 bytes 2024-09-14T23:07:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2742745088 bytes 2024-09-14T23:07:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2798918656 bytes 2024-09-14T23:07:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2802596864 bytes 2024-09-14T23:07:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2811806720 bytes 2024-09-14T23:07:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2821681152 bytes 2024-09-14T23:07:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:07:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:07:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:07:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:07:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_42 2024-09-14T23:07:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-14T23:07:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:07:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:07:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:07:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2024-09-14T23:07:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:07:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:07:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-14T23:07:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:07:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-14T23:07:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:07:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-14T23:07:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_42': 2024-09-14T23:07:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1409a9689ee4 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1409a9689ee4 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:36 2024 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:36 2024 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:36 2024 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 1dbaf776-62e8-4913-8034-899563d552d8 2024-09-14T23:07:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:07:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:07:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-14T23:07:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:07:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:07:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=13 2024-09-14T23:07:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:07:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:07:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 9128960 bytes 2024-09-14T23:07:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 41016320 bytes 2024-09-14T23:07:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 74958848 bytes 2024-09-14T23:07:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 114317312 bytes 2024-09-14T23:07:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 161509376 bytes 2024-09-14T23:07:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 214750208 bytes 2024-09-14T23:07:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 243661824 bytes 2024-09-14T23:07:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 262754304 bytes 2024-09-14T23:07:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 281555968 bytes 2024-09-14T23:07:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 316207104 bytes 2024-09-14T23:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 364732416 bytes 2024-09-14T23:07:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 395959296 bytes 2024-09-14T23:07:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 487455744 bytes 2024-09-14T23:07:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 539361280 bytes 2024-09-14T23:07:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 584053760 bytes 2024-09-14T23:07:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 620562432 bytes 2024-09-14T23:07:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 674017280 bytes 2024-09-14T23:07:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 775703552 bytes 2024-09-14T23:07:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 776302592 bytes 2024-09-14T23:07:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 826429440 bytes 2024-09-14T23:07:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 835787776 bytes 2024-09-14T23:07:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 882768896 bytes 2024-09-14T23:07:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 893609984 bytes 2024-09-14T23:07:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 939224064 bytes 2024-09-14T23:07:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 968201216 bytes 2024-09-14T23:07:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1003546624 bytes 2024-09-14T23:07:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1028322304 bytes 2024-09-14T23:07:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1052947456 bytes 2024-09-14T23:07:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1103671296 bytes 2024-09-14T23:07:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1124660224 bytes 2024-09-14T23:07:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1189070848 bytes 2024-09-14T23:07:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1202575360 bytes 2024-09-14T23:07:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1232610304 bytes 2024-09-14T23:07:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1259499520 bytes 2024-09-14T23:07:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1273734144 bytes 2024-09-14T23:07:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1362148352 bytes 2024-09-14T23:07:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1419488256 bytes 2024-09-14T23:07:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1488730112 bytes 2024-09-14T23:07:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1519395840 bytes 2024-09-14T23:07:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1543770112 bytes 2024-09-14T23:07:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1600801792 bytes 2024-09-14T23:07:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1651254272 bytes 2024-09-14T23:07:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1713520640 bytes 2024-09-14T23:07:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1745906688 bytes 2024-09-14T23:07:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1789032448 bytes 2024-09-14T23:07:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1924698112 bytes 2024-09-14T23:07:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1996225536 bytes 2024-09-14T23:07:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2045256704 bytes 2024-09-14T23:07:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2147840000 bytes 2024-09-14T23:07:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2238839808 bytes 2024-09-14T23:07:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2273630208 bytes 2024-09-14T23:07:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2348588032 bytes 2024-09-14T23:07:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2383947776 bytes 2024-09-14T23:07:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2426052608 bytes 2024-09-14T23:07:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2472437760 bytes 2024-09-14T23:07:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2474555392 bytes 2024-09-14T23:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2569201664 bytes 2024-09-14T23:07:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2614416384 bytes 2024-09-14T23:07:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2647257088 bytes 2024-09-14T23:07:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2717768704 bytes 2024-09-14T23:07:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2785252352 bytes 2024-09-14T23:07:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2830896128 bytes 2024-09-14T23:07:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2909581312 bytes 2024-09-14T23:07:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2980121600 bytes 2024-09-14T23:07:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 3046609920 bytes 2024-09-14T23:07:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 3135420416 bytes 2024-09-14T23:07:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 3161733120 bytes 2024-09-14T23:07:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 3181306880 bytes 2024-09-14T23:07:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3230840832 bytes 2024-09-14T23:07:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3266949120 bytes 2024-09-14T23:07:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3282752512 bytes 2024-09-14T23:07:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3357844480 bytes 2024-09-14T23:07:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3387625472 bytes 2024-09-14T23:07:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3411072000 bytes 2024-09-14T23:07:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3434516480 bytes 2024-09-14T23:07:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3475988480 bytes 2024-09-14T23:07:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3520448512 bytes 2024-09-14T23:07:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3565405184 bytes 2024-09-14T23:07:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3625799680 bytes 2024-09-14T23:07:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3695495168 bytes 2024-09-14T23:07:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3715835904 bytes 2024-09-14T23:07:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3721825280 bytes 2024-09-14T23:07:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3781444608 bytes 2024-09-14T23:07:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3810019328 bytes 2024-09-14T23:07:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3894141952 bytes 2024-09-14T23:07:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3910583296 bytes 2024-09-14T23:07:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3949787136 bytes 2024-09-14T23:07:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3975922688 bytes 2024-09-14T23:07:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 4022639616 bytes 2024-09-14T23:07:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 4047073280 bytes 2024-09-14T23:07:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 4069398528 bytes 2024-09-14T23:07:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 4093783040 bytes 2024-09-14T23:07:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 4143023104 bytes 2024-09-14T23:07:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4160976896 bytes 2024-09-14T23:07:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4214595584 bytes 2024-09-14T23:07:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4291568640 bytes 2024-09-14T23:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4344308736 bytes 2024-09-14T23:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4386727936 bytes 2024-09-14T23:07:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4441068544 bytes 2024-09-14T23:07:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4462451712 bytes 2024-09-14T23:07:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4485588992 bytes 2024-09-14T23:07:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4502135808 bytes 2024-09-14T23:07:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4519089152 bytes 2024-09-14T23:07:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4564831232 bytes 2024-09-14T23:07:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:07:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:07:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:07:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_43 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:07:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-14T23:07:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:07:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:07:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:07:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:07:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2024-09-14T23:07:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:07:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:07:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-14T23:07:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:07:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-14T23:07:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:07:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-14T23:07:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_43': 2024-09-14T23:07:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:07:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:07:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:07:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 14114bf48aa1 2024-09-14T23:07:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.14114bf48aa1 2024-09-14T23:07:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:07:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:07:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:07:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:07:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:50 2024 2024-09-14T23:07:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:50 2024 2024-09-14T23:07:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:50 2024 2024-09-14T23:07:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:07:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:07:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 800c56a4-4130-4ca1-a577-1c016136f961 2024-09-14T23:07:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:07:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:07:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-14T23:07:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:07:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:07:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T23:07:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:07:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:07:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 51201024 bytes 2024-09-14T23:07:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 73235456 bytes 2024-09-14T23:07:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 123686912 bytes 2024-09-14T23:07:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 187844608 bytes 2024-09-14T23:07:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 197076992 bytes 2024-09-14T23:07:52.023 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8355817100940235) > weight (0.25) 2024-09-14T23:07:52.023 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-14T23:07:52.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-14T23:07:52.023 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T23:07:52.024 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-14T23:07:52.024 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:07:52.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:07:52.023+0000 7f8626957640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 46599) UID: 1000 2024-09-14T23:07:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 227188736 bytes 2024-09-14T23:07:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 274145280 bytes 2024-09-14T23:07:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 348960768 bytes 2024-09-14T23:07:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 383798272 bytes 2024-09-14T23:07:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 385358848 bytes 2024-09-14T23:07:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 409710592 bytes 2024-09-14T23:07:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 446080000 bytes 2024-09-14T23:07:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 468013056 bytes 2024-09-14T23:07:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 539104256 bytes 2024-09-14T23:07:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 587413504 bytes 2024-09-14T23:07:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 665669632 bytes 2024-09-14T23:07:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 722980864 bytes 2024-09-14T23:07:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 770046976 bytes 2024-09-14T23:07:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 823754752 bytes 2024-09-14T23:07:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 841350144 bytes 2024-09-14T23:07:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 860681216 bytes 2024-09-14T23:07:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 879185920 bytes 2024-09-14T23:07:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 942756864 bytes 2024-09-14T23:07:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 972527616 bytes 2024-09-14T23:07:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1004948480 bytes 2024-09-14T23:07:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1005555712 bytes 2024-09-14T23:07:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1006168064 bytes 2024-09-14T23:07:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1068780544 bytes 2024-09-14T23:07:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1101565952 bytes 2024-09-14T23:07:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1125914624 bytes 2024-09-14T23:07:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1137069056 bytes 2024-09-14T23:07:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1165136896 bytes 2024-09-14T23:07:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1170516992 bytes 2024-09-14T23:07:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1243923456 bytes 2024-09-14T23:07:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1300604928 bytes 2024-09-14T23:07:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1337312256 bytes 2024-09-14T23:07:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1376852992 bytes 2024-09-14T23:07:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1416602624 bytes 2024-09-14T23:07:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1454157824 bytes 2024-09-14T23:07:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1542384640 bytes 2024-09-14T23:07:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1608103936 bytes 2024-09-14T23:07:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1638625280 bytes 2024-09-14T23:07:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1639236608 bytes 2024-09-14T23:07:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1705512960 bytes 2024-09-14T23:07:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1733523456 bytes 2024-09-14T23:07:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1764305920 bytes 2024-09-14T23:07:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1802243072 bytes 2024-09-14T23:07:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1811879936 bytes 2024-09-14T23:07:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1922222080 bytes 2024-09-14T23:07:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1969829888 bytes 2024-09-14T23:07:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:07:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:07:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:07:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_44 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-14T23:07:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:07:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:07:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:07:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:07:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:07:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2024-09-14T23:07:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:07:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:07:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:07:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-14T23:07:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:07:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:07:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-14T23:07:58.125 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-14T23:07:58.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T23:07:58.126 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T23:07:58.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T23:07:58.129 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-09-14T23:07:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:07:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-14T23:07:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_44': 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 141b870427bf 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.141b870427bf 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:07:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:57 2024 2024-09-14T23:07:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:57 2024 2024-09-14T23:07:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:57 2024 2024-09-14T23:07:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:07:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:07:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 1657a08a-2ee0-4427-8918-43d3fca35d43 2024-09-14T23:07:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:07:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:07:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-14T23:07:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:07:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:07:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:07:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:07:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=14 2024-09-14T23:07:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:07:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:07:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 24574976 bytes 2024-09-14T23:07:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 61893632 bytes 2024-09-14T23:07:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 139764736 bytes 2024-09-14T23:07:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 171193344 bytes 2024-09-14T23:07:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 188960768 bytes 2024-09-14T23:07:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 283828224 bytes 2024-09-14T23:07:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 298041344 bytes 2024-09-14T23:07:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 312948736 bytes 2024-09-14T23:07:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 328551424 bytes 2024-09-14T23:07:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 397419520 bytes 2024-09-14T23:07:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 449544192 bytes 2024-09-14T23:07:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 528667648 bytes 2024-09-14T23:07:59.918 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7948586600783432) > weight (0.25) 2024-09-14T23:07:59.918 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4804723190828494) > weight (0.25) 2024-09-14T23:07:59.919 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-14T23:07:59.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-14T23:07:59.919 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-14T23:07:59.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:07:59.918+0000 7f37e7918640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 47556) UID: 1000 2024-09-14T23:08:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 588374016 bytes 2024-09-14T23:08:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 605155328 bytes 2024-09-14T23:08:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 672438272 bytes 2024-09-14T23:08:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 703171584 bytes 2024-09-14T23:08:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 716924928 bytes 2024-09-14T23:08:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 781335552 bytes 2024-09-14T23:08:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 806751232 bytes 2024-09-14T23:08:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 825415680 bytes 2024-09-14T23:08:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 829199360 bytes 2024-09-14T23:08:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 835226624 bytes 2024-09-14T23:08:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 881073152 bytes 2024-09-14T23:08:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 882308096 bytes 2024-09-14T23:08:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 919230464 bytes 2024-09-14T23:08:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1002804224 bytes 2024-09-14T23:08:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1027855360 bytes 2024-09-14T23:08:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1055262720 bytes 2024-09-14T23:08:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1115286528 bytes 2024-09-14T23:08:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1149086720 bytes 2024-09-14T23:08:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1200043008 bytes 2024-09-14T23:08:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1254820864 bytes 2024-09-14T23:08:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1284710400 bytes 2024-09-14T23:08:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1345770496 bytes 2024-09-14T23:08:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1397748736 bytes 2024-09-14T23:08:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1472152576 bytes 2024-09-14T23:08:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1494757376 bytes 2024-09-14T23:08:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1551574016 bytes 2024-09-14T23:08:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1552169984 bytes 2024-09-14T23:08:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1552768000 bytes 2024-09-14T23:08:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1562461184 bytes 2024-09-14T23:08:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1583249408 bytes 2024-09-14T23:08:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1604786176 bytes 2024-09-14T23:08:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1741967360 bytes 2024-09-14T23:08:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1813312512 bytes 2024-09-14T23:08:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1903447040 bytes 2024-09-14T23:08:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1920770048 bytes 2024-09-14T23:08:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1983323136 bytes 2024-09-14T23:08:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1983941632 bytes 2024-09-14T23:08:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1993681920 bytes 2024-09-14T23:08:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2018813952 bytes 2024-09-14T23:08:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2063695872 bytes 2024-09-14T23:08:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2101165056 bytes 2024-09-14T23:08:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2168001536 bytes 2024-09-14T23:08:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2204822528 bytes 2024-09-14T23:08:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2205438976 bytes 2024-09-14T23:08:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2275141632 bytes 2024-09-14T23:08:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2309850112 bytes 2024-09-14T23:08:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2310475776 bytes 2024-09-14T23:08:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2365206528 bytes 2024-09-14T23:08:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2381775872 bytes 2024-09-14T23:08:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2445497344 bytes 2024-09-14T23:08:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2518982656 bytes 2024-09-14T23:08:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2519589888 bytes 2024-09-14T23:08:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2570328064 bytes 2024-09-14T23:08:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2656926720 bytes 2024-09-14T23:08:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2689387520 bytes 2024-09-14T23:08:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2746209280 bytes 2024-09-14T23:08:06.920 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-14T23:08:06.920 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:08:06.921 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-14T23:08:06.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T23:08:06.921 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T23:08:06.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T23:08:06.925 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-09-14T23:08:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2785390592 bytes 2024-09-14T23:08:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2787491840 bytes 2024-09-14T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2801951744 bytes 2024-09-14T23:08:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2897138688 bytes 2024-09-14T23:08:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2948801536 bytes 2024-09-14T23:08:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2975740928 bytes 2024-09-14T23:08:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3021668352 bytes 2024-09-14T23:08:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3115053056 bytes 2024-09-14T23:08:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3139976192 bytes 2024-09-14T23:08:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3160083456 bytes 2024-09-14T23:08:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3218894848 bytes 2024-09-14T23:08:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3296166912 bytes 2024-09-14T23:08:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3336565760 bytes 2024-09-14T23:08:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3376126976 bytes 2024-09-14T23:08:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3428071424 bytes 2024-09-14T23:08:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3496199168 bytes 2024-09-14T23:08:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3522285568 bytes 2024-09-14T23:08:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3538494464 bytes 2024-09-14T23:08:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3561708544 bytes 2024-09-14T23:08:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3577016320 bytes 2024-09-14T23:08:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3595622400 bytes 2024-09-14T23:08:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3633833984 bytes 2024-09-14T23:08:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3634440192 bytes 2024-09-14T23:08:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3697537024 bytes 2024-09-14T23:08:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3742337024 bytes 2024-09-14T23:08:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3790638080 bytes 2024-09-14T23:08:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 3826192384 bytes 2024-09-14T23:08:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 3881126912 bytes 2024-09-14T23:08:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 3900053504 bytes 2024-09-14T23:08:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 3953513472 bytes 2024-09-14T23:08:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 3972021248 bytes 2024-09-14T23:08:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4018722816 bytes 2024-09-14T23:08:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4049183744 bytes 2024-09-14T23:08:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4058400768 bytes 2024-09-14T23:08:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4079812608 bytes 2024-09-14T23:08:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4139831296 bytes 2024-09-14T23:08:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4231550976 bytes 2024-09-14T23:08:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4278333440 bytes 2024-09-14T23:08:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4319937536 bytes 2024-09-14T23:08:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4361694208 bytes 2024-09-14T23:08:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 4453167104 bytes 2024-09-14T23:08:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11000 IOs, 4474402816 bytes 2024-09-14T23:08:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11100 IOs, 4539374592 bytes 2024-09-14T23:08:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11200 IOs, 4560933888 bytes 2024-09-14T23:08:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:08:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:08:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:08:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:08:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_45 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-14T23:08:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2024-09-14T23:08:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:08:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:08:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:08:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-14T23:08:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:08:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:08:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-14T23:08:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:08:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-14T23:08:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_45': 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1432de0045b 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1432de0045b 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:12 2024 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:12 2024 2024-09-14T23:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:12 2024 2024-09-14T23:08:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:08:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:08:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 384c1652-4afa-4fe7-8287-d17f60057ca1 2024-09-14T23:08:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:08:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:08:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-14T23:08:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:08:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:08:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=10 2024-09-14T23:08:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:08:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:08:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 47583232 bytes 2024-09-14T23:08:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 51261440 bytes 2024-09-14T23:08:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 126184448 bytes 2024-09-14T23:08:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 151256064 bytes 2024-09-14T23:08:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 197337088 bytes 2024-09-14T23:08:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 197943296 bytes 2024-09-14T23:08:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 236457984 bytes 2024-09-14T23:08:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 277949440 bytes 2024-09-14T23:08:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 309689344 bytes 2024-09-14T23:08:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 393055232 bytes 2024-09-14T23:08:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 432263168 bytes 2024-09-14T23:08:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 437023744 bytes 2024-09-14T23:08:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 539535360 bytes 2024-09-14T23:08:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 540151808 bytes 2024-09-14T23:08:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 554979328 bytes 2024-09-14T23:08:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 586641408 bytes 2024-09-14T23:08:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 667833344 bytes 2024-09-14T23:08:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 696268800 bytes 2024-09-14T23:08:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 746484736 bytes 2024-09-14T23:08:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 784182272 bytes 2024-09-14T23:08:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 826146816 bytes 2024-09-14T23:08:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 847680512 bytes 2024-09-14T23:08:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 882810880 bytes 2024-09-14T23:08:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 912799744 bytes 2024-09-14T23:08:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 987958272 bytes 2024-09-14T23:08:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1098756096 bytes 2024-09-14T23:08:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1160235008 bytes 2024-09-14T23:08:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1180052480 bytes 2024-09-14T23:08:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1252929536 bytes 2024-09-14T23:08:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1284281344 bytes 2024-09-14T23:08:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1379165184 bytes 2024-09-14T23:08:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1404697600 bytes 2024-09-14T23:08:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1417383936 bytes 2024-09-14T23:08:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1448298496 bytes 2024-09-14T23:08:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1507471360 bytes 2024-09-14T23:08:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1561207808 bytes 2024-09-14T23:08:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1609424896 bytes 2024-09-14T23:08:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1674692608 bytes 2024-09-14T23:08:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1699693568 bytes 2024-09-14T23:08:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1769394176 bytes 2024-09-14T23:08:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1820028928 bytes 2024-09-14T23:08:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1901498368 bytes 2024-09-14T23:08:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1927029760 bytes 2024-09-14T23:08:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1975839744 bytes 2024-09-14T23:08:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2045862912 bytes 2024-09-14T23:08:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2046486528 bytes 2024-09-14T23:08:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2047111168 bytes 2024-09-14T23:08:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2089276416 bytes 2024-09-14T23:08:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2157028352 bytes 2024-09-14T23:08:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2195597312 bytes 2024-09-14T23:08:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2240322560 bytes 2024-09-14T23:08:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2275215360 bytes 2024-09-14T23:08:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2334272512 bytes 2024-09-14T23:08:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2347268096 bytes 2024-09-14T23:08:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2384722944 bytes 2024-09-14T23:08:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2428242944 bytes 2024-09-14T23:08:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2473911296 bytes 2024-09-14T23:08:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2510397440 bytes 2024-09-14T23:08:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2511020032 bytes 2024-09-14T23:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2529741824 bytes 2024-09-14T23:08:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2587729920 bytes 2024-09-14T23:08:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2660409344 bytes 2024-09-14T23:08:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2685504512 bytes 2024-09-14T23:08:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2749420544 bytes 2024-09-14T23:08:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2803153920 bytes 2024-09-14T23:08:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2887012352 bytes 2024-09-14T23:08:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2931997696 bytes 2024-09-14T23:08:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2978079744 bytes 2024-09-14T23:08:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3003634688 bytes 2024-09-14T23:08:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3036660736 bytes 2024-09-14T23:08:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3087752192 bytes 2024-09-14T23:08:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3121661952 bytes 2024-09-14T23:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3140085760 bytes 2024-09-14T23:08:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3188538368 bytes 2024-09-14T23:08:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3261413376 bytes 2024-09-14T23:08:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3286075392 bytes 2024-09-14T23:08:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3324086272 bytes 2024-09-14T23:08:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3385394176 bytes 2024-09-14T23:08:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3405012992 bytes 2024-09-14T23:08:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3407705088 bytes 2024-09-14T23:08:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3436091392 bytes 2024-09-14T23:08:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3507617792 bytes 2024-09-14T23:08:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:08:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:08:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_46 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-14T23:08:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2024-09-14T23:08:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:08:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:08:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:08:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-14T23:08:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:08:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:08:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-14T23:08:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:08:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-14T23:08:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_46': 2024-09-14T23:08:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:08:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 143d89830757 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.143d89830757 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:23 2024 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:23 2024 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:23 2024 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: fbfee603-c1c3-48f3-b0df-17ec0d3552e7 2024-09-14T23:08:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:08:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:08:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-14T23:08:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:08:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:08:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=9 2024-09-14T23:08:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:08:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:08:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 18251776 bytes 2024-09-14T23:08:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 18882560 bytes 2024-09-14T23:08:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 49231872 bytes 2024-09-14T23:08:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 94503936 bytes 2024-09-14T23:08:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 118174720 bytes 2024-09-14T23:08:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 186951680 bytes 2024-09-14T23:08:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 233959424 bytes 2024-09-14T23:08:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 298773504 bytes 2024-09-14T23:08:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 340347904 bytes 2024-09-14T23:08:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 392232960 bytes 2024-09-14T23:08:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 415490048 bytes 2024-09-14T23:08:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 458671104 bytes 2024-09-14T23:08:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 480050176 bytes 2024-09-14T23:08:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 488858624 bytes 2024-09-14T23:08:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 516602880 bytes 2024-09-14T23:08:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 556434432 bytes 2024-09-14T23:08:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 606231552 bytes 2024-09-14T23:08:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 681118720 bytes 2024-09-14T23:08:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 735133696 bytes 2024-09-14T23:08:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 830017536 bytes 2024-09-14T23:08:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 866294784 bytes 2024-09-14T23:08:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 902418432 bytes 2024-09-14T23:08:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 921003008 bytes 2024-09-14T23:08:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1014369280 bytes 2024-09-14T23:08:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1065586688 bytes 2024-09-14T23:08:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1104279552 bytes 2024-09-14T23:08:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1122404352 bytes 2024-09-14T23:08:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1138322432 bytes 2024-09-14T23:08:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1203827712 bytes 2024-09-14T23:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1218065408 bytes 2024-09-14T23:08:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1222922240 bytes 2024-09-14T23:08:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1314899968 bytes 2024-09-14T23:08:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1388048384 bytes 2024-09-14T23:08:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1470323712 bytes 2024-09-14T23:08:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1514052608 bytes 2024-09-14T23:08:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1580150784 bytes 2024-09-14T23:08:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1620083712 bytes 2024-09-14T23:08:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1729474560 bytes 2024-09-14T23:08:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1777627136 bytes 2024-09-14T23:08:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1841823744 bytes 2024-09-14T23:08:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1859410944 bytes 2024-09-14T23:08:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1868706816 bytes 2024-09-14T23:08:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1899523072 bytes 2024-09-14T23:08:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1978773504 bytes 2024-09-14T23:08:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2028578816 bytes 2024-09-14T23:08:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2097803264 bytes 2024-09-14T23:08:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2136699904 bytes 2024-09-14T23:08:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2198504448 bytes 2024-09-14T23:08:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2208033792 bytes 2024-09-14T23:08:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2281600000 bytes 2024-09-14T23:08:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2320248832 bytes 2024-09-14T23:08:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2392481792 bytes 2024-09-14T23:08:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2417094656 bytes 2024-09-14T23:08:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2486554624 bytes 2024-09-14T23:08:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2536196096 bytes 2024-09-14T23:08:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2619757568 bytes 2024-09-14T23:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2634135552 bytes 2024-09-14T23:08:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2689691648 bytes 2024-09-14T23:08:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2732476416 bytes 2024-09-14T23:08:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2772521984 bytes 2024-09-14T23:08:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2797132800 bytes 2024-09-14T23:08:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2842287104 bytes 2024-09-14T23:08:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2900166656 bytes 2024-09-14T23:08:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2929977344 bytes 2024-09-14T23:08:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2951624704 bytes 2024-09-14T23:08:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 3031096320 bytes 2024-09-14T23:08:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 3059208192 bytes 2024-09-14T23:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 3059825664 bytes 2024-09-14T23:08:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3130717184 bytes 2024-09-14T23:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3158196224 bytes 2024-09-14T23:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3181981696 bytes 2024-09-14T23:08:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:08:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:08:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:08:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:08:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:08:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_47 2024-09-14T23:08:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:08:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:08:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:08:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:08:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2024-09-14T23:08:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:08:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:08:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:08:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-14T23:08:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:08:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:08:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-14T23:08:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:08:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-14T23:08:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_47': 2024-09-14T23:08:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:08:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:08:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:08:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 144655a96e0 2024-09-14T23:08:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.144655a96e0 2024-09-14T23:08:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:33 2024 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:33 2024 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:33 2024 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 9dc01f72-5b4f-4251-988c-6e2524bebd28 2024-09-14T23:08:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:08:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:08:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-14T23:08:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:08:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:08:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=12 2024-09-14T23:08:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:08:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:08:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 16923648 bytes 2024-09-14T23:08:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 80116736 bytes 2024-09-14T23:08:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 119879680 bytes 2024-09-14T23:08:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 193849344 bytes 2024-09-14T23:08:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 246190080 bytes 2024-09-14T23:08:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 318273536 bytes 2024-09-14T23:08:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 388187136 bytes 2024-09-14T23:08:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 398910464 bytes 2024-09-14T23:08:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 467663872 bytes 2024-09-14T23:08:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 567385088 bytes 2024-09-14T23:08:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 587154432 bytes 2024-09-14T23:08:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 630044672 bytes 2024-09-14T23:08:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 684778496 bytes 2024-09-14T23:08:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 743688192 bytes 2024-09-14T23:08:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 746341376 bytes 2024-09-14T23:08:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 835014656 bytes 2024-09-14T23:08:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 852238336 bytes 2024-09-14T23:08:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 861100032 bytes 2024-09-14T23:08:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 911197184 bytes 2024-09-14T23:08:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 927583232 bytes 2024-09-14T23:08:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 938774528 bytes 2024-09-14T23:08:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 986906624 bytes 2024-09-14T23:08:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1005217792 bytes 2024-09-14T23:08:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1025615872 bytes 2024-09-14T23:08:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1078001664 bytes 2024-09-14T23:08:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1150896128 bytes 2024-09-14T23:08:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1192140800 bytes 2024-09-14T23:08:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1237120000 bytes 2024-09-14T23:08:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1254406144 bytes 2024-09-14T23:08:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1327574016 bytes 2024-09-14T23:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1343998976 bytes 2024-09-14T23:08:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1367156736 bytes 2024-09-14T23:08:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1400902656 bytes 2024-09-14T23:08:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1415024640 bytes 2024-09-14T23:08:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1481182208 bytes 2024-09-14T23:08:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1494864896 bytes 2024-09-14T23:08:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1522159616 bytes 2024-09-14T23:08:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1546804224 bytes 2024-09-14T23:08:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1572558848 bytes 2024-09-14T23:08:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1640344576 bytes 2024-09-14T23:08:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1686921216 bytes 2024-09-14T23:08:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1754460160 bytes 2024-09-14T23:08:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1772685312 bytes 2024-09-14T23:08:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1839101952 bytes 2024-09-14T23:08:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1856071680 bytes 2024-09-14T23:08:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1885352960 bytes 2024-09-14T23:08:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1939656704 bytes 2024-09-14T23:08:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1991034880 bytes 2024-09-14T23:08:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2046902272 bytes 2024-09-14T23:08:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2093396992 bytes 2024-09-14T23:08:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2141033472 bytes 2024-09-14T23:08:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2200830976 bytes 2024-09-14T23:08:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2245655552 bytes 2024-09-14T23:08:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2261614592 bytes 2024-09-14T23:08:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2276664320 bytes 2024-09-14T23:08:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2301663232 bytes 2024-09-14T23:08:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2342561792 bytes 2024-09-14T23:08:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2391285760 bytes 2024-09-14T23:08:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2426608640 bytes 2024-09-14T23:08:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2487376896 bytes 2024-09-14T23:08:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2527026176 bytes 2024-09-14T23:08:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2549728256 bytes 2024-09-14T23:08:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2559438848 bytes 2024-09-14T23:08:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2652879872 bytes 2024-09-14T23:08:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2658938880 bytes 2024-09-14T23:08:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2718822400 bytes 2024-09-14T23:08:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2719689728 bytes 2024-09-14T23:08:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2748140544 bytes 2024-09-14T23:08:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2766757888 bytes 2024-09-14T23:08:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2820673536 bytes 2024-09-14T23:08:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2921852928 bytes 2024-09-14T23:08:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2984281088 bytes 2024-09-14T23:08:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3026743296 bytes 2024-09-14T23:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3079994368 bytes 2024-09-14T23:08:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3127191552 bytes 2024-09-14T23:08:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3194563584 bytes 2024-09-14T23:08:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3238593536 bytes 2024-09-14T23:08:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3258438656 bytes 2024-09-14T23:08:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3299001344 bytes 2024-09-14T23:08:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3335726080 bytes 2024-09-14T23:08:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3386398720 bytes 2024-09-14T23:08:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3408095232 bytes 2024-09-14T23:08:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3439873024 bytes 2024-09-14T23:08:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3487727616 bytes 2024-09-14T23:08:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3517254656 bytes 2024-09-14T23:08:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3568855040 bytes 2024-09-14T23:08:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3664625664 bytes 2024-09-14T23:08:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3701354496 bytes 2024-09-14T23:08:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3788376064 bytes 2024-09-14T23:08:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3802791936 bytes 2024-09-14T23:08:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3868739584 bytes 2024-09-14T23:08:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3895490560 bytes 2024-09-14T23:08:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3932119040 bytes 2024-09-14T23:08:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4005820416 bytes 2024-09-14T23:08:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4063666176 bytes 2024-09-14T23:08:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4064282624 bytes 2024-09-14T23:08:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_48 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:08:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:08:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:08:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2024-09-14T23:08:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:08:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:08:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:08:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-14T23:08:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:08:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:08:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-14T23:08:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:08:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-14T23:08:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_48': 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 144e43200894 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.144e43200894 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:46 2024 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:46 2024 2024-09-14T23:08:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:46 2024 2024-09-14T23:08:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:08:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:08:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 11b0a986-da1c-46e1-bd44-b5b3d1b231b5 2024-09-14T23:08:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:08:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:08:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-14T23:08:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:08:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:08:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:08:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:08:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=13 2024-09-14T23:08:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:08:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:08:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 611328 bytes 2024-09-14T23:08:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 35940352 bytes 2024-09-14T23:08:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 91326464 bytes 2024-09-14T23:08:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 91944960 bytes 2024-09-14T23:08:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 120689664 bytes 2024-09-14T23:08:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 158948352 bytes 2024-09-14T23:08:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 176114688 bytes 2024-09-14T23:08:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 228434944 bytes 2024-09-14T23:08:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 259136512 bytes 2024-09-14T23:08:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 259733504 bytes 2024-09-14T23:08:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 314532864 bytes 2024-09-14T23:08:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 383378432 bytes 2024-09-14T23:08:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 434856960 bytes 2024-09-14T23:08:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 506473472 bytes 2024-09-14T23:08:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 588490752 bytes 2024-09-14T23:08:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 598648832 bytes 2024-09-14T23:08:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 616295424 bytes 2024-09-14T23:08:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 673077248 bytes 2024-09-14T23:08:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 711676928 bytes 2024-09-14T23:08:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 813558784 bytes 2024-09-14T23:08:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 867644416 bytes 2024-09-14T23:08:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 892420096 bytes 2024-09-14T23:08:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 947002368 bytes 2024-09-14T23:08:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1016297472 bytes 2024-09-14T23:08:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1035923456 bytes 2024-09-14T23:08:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1036530688 bytes 2024-09-14T23:08:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1062086656 bytes 2024-09-14T23:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1073057792 bytes 2024-09-14T23:08:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1118271488 bytes 2024-09-14T23:08:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1173443584 bytes 2024-09-14T23:08:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1225737216 bytes 2024-09-14T23:08:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1226587136 bytes 2024-09-14T23:08:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1237781504 bytes 2024-09-14T23:08:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1240902656 bytes 2024-09-14T23:08:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1271215104 bytes 2024-09-14T23:08:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1339423744 bytes 2024-09-14T23:08:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1355625472 bytes 2024-09-14T23:08:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1394549760 bytes 2024-09-14T23:08:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1418671104 bytes 2024-09-14T23:08:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1424622592 bytes 2024-09-14T23:08:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1488702464 bytes 2024-09-14T23:08:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1492454400 bytes 2024-09-14T23:08:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1579231232 bytes 2024-09-14T23:08:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1650272256 bytes 2024-09-14T23:08:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1717627904 bytes 2024-09-14T23:08:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1776029696 bytes 2024-09-14T23:08:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1869034496 bytes 2024-09-14T23:08:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1939519488 bytes 2024-09-14T23:08:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1995324416 bytes 2024-09-14T23:08:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2045854720 bytes 2024-09-14T23:08:53.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2069664768 bytes 2024-09-14T23:08:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2070242304 bytes 2024-09-14T23:08:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2129398784 bytes 2024-09-14T23:08:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2169521152 bytes 2024-09-14T23:08:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2204237824 bytes 2024-09-14T23:08:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2238312448 bytes 2024-09-14T23:08:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2246155264 bytes 2024-09-14T23:08:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2284937216 bytes 2024-09-14T23:08:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2333897728 bytes 2024-09-14T23:08:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2420805632 bytes 2024-09-14T23:08:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2461883392 bytes 2024-09-14T23:08:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2495517696 bytes 2024-09-14T23:08:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2499129344 bytes 2024-09-14T23:08:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2529222656 bytes 2024-09-14T23:08:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2548026368 bytes 2024-09-14T23:08:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2565839872 bytes 2024-09-14T23:08:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2641214464 bytes 2024-09-14T23:08:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2662433792 bytes 2024-09-14T23:08:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2706170880 bytes 2024-09-14T23:08:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2743737344 bytes 2024-09-14T23:08:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2744354816 bytes 2024-09-14T23:08:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2808538112 bytes 2024-09-14T23:08:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 2831966208 bytes 2024-09-14T23:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 2932135936 bytes 2024-09-14T23:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 2983270400 bytes 2024-09-14T23:08:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3034459136 bytes 2024-09-14T23:08:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3036027904 bytes 2024-09-14T23:08:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3059972096 bytes 2024-09-14T23:08:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3179546624 bytes 2024-09-14T23:08:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3225090048 bytes 2024-09-14T23:08:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3248897024 bytes 2024-09-14T23:08:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3259930624 bytes 2024-09-14T23:08:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3347215360 bytes 2024-09-14T23:08:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3408201728 bytes 2024-09-14T23:08:57.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3491108864 bytes 2024-09-14T23:08:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3591193600 bytes 2024-09-14T23:08:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3604288512 bytes 2024-09-14T23:08:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3664438272 bytes 2024-09-14T23:08:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3665060864 bytes 2024-09-14T23:08:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3715410944 bytes 2024-09-14T23:08:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3749575680 bytes 2024-09-14T23:08:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3845487616 bytes 2024-09-14T23:08:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3893115904 bytes 2024-09-14T23:08:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3911003136 bytes 2024-09-14T23:08:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 3967216640 bytes 2024-09-14T23:08:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4037548032 bytes 2024-09-14T23:08:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4109929472 bytes 2024-09-14T23:08:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4138090496 bytes 2024-09-14T23:08:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4175151104 bytes 2024-09-14T23:08:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4198696960 bytes 2024-09-14T23:08:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4241863680 bytes 2024-09-14T23:08:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4259761152 bytes 2024-09-14T23:08:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4313175040 bytes 2024-09-14T23:09:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4346691584 bytes 2024-09-14T23:09:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4395508736 bytes 2024-09-14T23:09:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4455322624 bytes 2024-09-14T23:09:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4455917568 bytes 2024-09-14T23:09:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4473187328 bytes 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_49 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:09:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:09:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:09:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2024-09-14T23:09:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:09:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:09:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-14T23:09:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:09:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:09:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-14T23:09:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:09:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-14T23:09:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_49': 2024-09-14T23:09:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:09:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:09:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 145633e2cde1 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.145633e2cde1 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:09:00 2024 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:09:00 2024 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:09:00 2024 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 909132f1-ded4-4f35-97e5-a59b71439534 2024-09-14T23:09:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:09:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:09:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-14T23:09:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:09:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:09:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=8 2024-09-14T23:09:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:09:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:09:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 62743552 bytes 2024-09-14T23:09:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 142939136 bytes 2024-09-14T23:09:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 179212288 bytes 2024-09-14T23:09:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 250844160 bytes 2024-09-14T23:09:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 293395456 bytes 2024-09-14T23:09:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 294010880 bytes 2024-09-14T23:09:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 329418752 bytes 2024-09-14T23:09:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 380723200 bytes 2024-09-14T23:09:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 381347840 bytes 2024-09-14T23:09:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 423344128 bytes 2024-09-14T23:09:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 450956288 bytes 2024-09-14T23:09:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 491685888 bytes 2024-09-14T23:09:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 530062336 bytes 2024-09-14T23:09:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 536279040 bytes 2024-09-14T23:09:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 551814144 bytes 2024-09-14T23:09:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 570110976 bytes 2024-09-14T23:09:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 619735040 bytes 2024-09-14T23:09:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 651205632 bytes 2024-09-14T23:09:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 746037248 bytes 2024-09-14T23:09:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 802801664 bytes 2024-09-14T23:09:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 848803840 bytes 2024-09-14T23:09:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 880319488 bytes 2024-09-14T23:09:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 939080704 bytes 2024-09-14T23:09:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 998509568 bytes 2024-09-14T23:09:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1029176320 bytes 2024-09-14T23:09:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1054310400 bytes 2024-09-14T23:09:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1082805248 bytes 2024-09-14T23:09:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1213832192 bytes 2024-09-14T23:09:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1282580480 bytes 2024-09-14T23:09:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1324794880 bytes 2024-09-14T23:09:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1390544896 bytes 2024-09-14T23:09:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1442790400 bytes 2024-09-14T23:09:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1504434176 bytes 2024-09-14T23:09:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1555484672 bytes 2024-09-14T23:09:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1570044928 bytes 2024-09-14T23:09:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1595508736 bytes 2024-09-14T23:09:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1622266880 bytes 2024-09-14T23:09:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1675939840 bytes 2024-09-14T23:09:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1679556608 bytes 2024-09-14T23:09:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1780345856 bytes 2024-09-14T23:09:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1788102656 bytes 2024-09-14T23:09:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1807770624 bytes 2024-09-14T23:09:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1850582016 bytes 2024-09-14T23:09:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1874266112 bytes 2024-09-14T23:09:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1904761856 bytes 2024-09-14T23:09:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1975798784 bytes 2024-09-14T23:09:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2004798464 bytes 2024-09-14T23:09:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2052888576 bytes 2024-09-14T23:09:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2102545408 bytes 2024-09-14T23:09:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2139175936 bytes 2024-09-14T23:09:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2139807744 bytes 2024-09-14T23:09:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2173235200 bytes 2024-09-14T23:09:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2205211648 bytes 2024-09-14T23:09:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2257002496 bytes 2024-09-14T23:09:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2316992512 bytes 2024-09-14T23:09:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2333079552 bytes 2024-09-14T23:09:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2350712832 bytes 2024-09-14T23:09:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2380880896 bytes 2024-09-14T23:09:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2441085952 bytes 2024-09-14T23:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2478928896 bytes 2024-09-14T23:09:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2576221184 bytes 2024-09-14T23:09:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2614474752 bytes 2024-09-14T23:09:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2694367232 bytes 2024-09-14T23:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2704447488 bytes 2024-09-14T23:09:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2761239552 bytes 2024-09-14T23:09:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:09:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_50 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:09:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n snapshot ']' 2024-09-14T23:09:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-14T23:09:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:09:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:09:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 128M ']' 2024-09-14T23:09:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=128M 2024-09-14T23:09:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:09:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2024-09-14T23:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T23:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' image = image ']' 2024-09-14T23:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-14T23:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:09:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:09:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-14T23:09:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:09:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-14T23:09:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_50': 2024-09-14T23:09:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 145fbd59c196 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.145fbd59c196 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:09:09 2024 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:09:09 2024 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:09:09 2024 2024-09-14T23:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:09:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:09:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: af809fea-94db-4aaf-b996-c34cbeff8227 2024-09-14T23:09:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:09:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n '' ']' 2024-09-14T23:09:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-14T23:09:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:09:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:09:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=7 2024-09-14T23:09:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:09:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:09:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 55050240 bytes 2024-09-14T23:09:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 89178112 bytes 2024-09-14T23:09:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 122259456 bytes 2024-09-14T23:09:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 147397632 bytes 2024-09-14T23:09:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 189604864 bytes 2024-09-14T23:09:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 215683072 bytes 2024-09-14T23:09:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 268405760 bytes 2024-09-14T23:09:11.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 312379392 bytes 2024-09-14T23:09:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 339253248 bytes 2024-09-14T23:09:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 339841024 bytes 2024-09-14T23:09:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 373699584 bytes 2024-09-14T23:09:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 393779200 bytes 2024-09-14T23:09:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 432939008 bytes 2024-09-14T23:09:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 487990272 bytes 2024-09-14T23:09:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 529350656 bytes 2024-09-14T23:09:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 556938240 bytes 2024-09-14T23:09:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 590737408 bytes 2024-09-14T23:09:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 644984832 bytes 2024-09-14T23:09:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 708817920 bytes 2024-09-14T23:09:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 720155648 bytes 2024-09-14T23:09:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 758546432 bytes 2024-09-14T23:09:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 814202880 bytes 2024-09-14T23:09:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 822850560 bytes 2024-09-14T23:09:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 855997440 bytes 2024-09-14T23:09:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 915124224 bytes 2024-09-14T23:09:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 937222144 bytes 2024-09-14T23:09:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 973224960 bytes 2024-09-14T23:09:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1002000384 bytes 2024-09-14T23:09:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1048924160 bytes 2024-09-14T23:09:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1068977152 bytes 2024-09-14T23:09:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1143493632 bytes 2024-09-14T23:09:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1201202176 bytes 2024-09-14T23:09:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1201823744 bytes 2024-09-14T23:09:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1222956032 bytes 2024-09-14T23:09:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1252408320 bytes 2024-09-14T23:09:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1310795776 bytes 2024-09-14T23:09:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1324472320 bytes 2024-09-14T23:09:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1343634432 bytes 2024-09-14T23:09:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1376962560 bytes 2024-09-14T23:09:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1433625600 bytes 2024-09-14T23:09:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1461089280 bytes 2024-09-14T23:09:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1505795072 bytes 2024-09-14T23:09:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1554531328 bytes 2024-09-14T23:09:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1623554048 bytes 2024-09-14T23:09:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1687694336 bytes 2024-09-14T23:09:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1709931520 bytes 2024-09-14T23:09:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1725168640 bytes 2024-09-14T23:09:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1738228736 bytes 2024-09-14T23:09:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1765416960 bytes 2024-09-14T23:09:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1793094656 bytes 2024-09-14T23:09:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 1801728000 bytes 2024-09-14T23:09:16.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 1802316800 bytes 2024-09-14T23:09:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 1837694976 bytes 2024-09-14T23:09:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 1839073280 bytes 2024-09-14T23:09:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 1857971200 bytes 2024-09-14T23:09:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 1990754304 bytes 2024-09-14T23:09:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2014451712 bytes 2024-09-14T23:09:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2062566400 bytes 2024-09-14T23:09:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2093026304 bytes 2024-09-14T23:09:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:09:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:09:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:09:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-14T23:09:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_count=50 2024-09-14T23:09:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local count 2024-09-14T23:09:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_count=0 2024-09-14T23:09:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 40 2024-09-14T23:09:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in `seq 1 40` 2024-09-14T23:09:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 1 -ne 1 2024-09-14T23:09:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-14T23:09:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep 'images: ' 2024-09-14T23:09:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cut '-d ' -f 2 2024-09-14T23:09:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T23:09:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ count=50 2024-09-14T23:09:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 50 = 50 2024-09-14T23:09:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-14T23:09:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state 2024-09-14T23:09:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 40 2024-09-14T23:09:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in `seq 1 40` 2024-09-14T23:09:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 1 -ne 1 2024-09-14T23:09:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-14T23:09:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep 'image health:' 2024-09-14T23:09:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cut '-d ' -f 3 2024-09-14T23:09:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=OK 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test OK = ERROR 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test OK = OK 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_count=50 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local count 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_count=0 2024-09-14T23:09:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 40 2024-09-14T23:09:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in `seq 1 40` 2024-09-14T23:09:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 1 -ne 1 2024-09-14T23:09:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T23:09:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep 'images: ' 2024-09-14T23:09:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cut '-d ' -f 2 2024-09-14T23:09:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T23:09:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ count=50 2024-09-14T23:09:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 50 = 50 2024-09-14T23:09:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-14T23:09:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state 2024-09-14T23:09:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 40 2024-09-14T23:09:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in `seq 1 40` 2024-09-14T23:09:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 1 -ne 1 2024-09-14T23:09:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T23:09:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep 'image health:' 2024-09-14T23:09:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cut '-d ' -f 3 2024-09-14T23:09:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T23:09:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=OK 2024-09-14T23:09:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test OK = ERROR 2024-09-14T23:09:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test OK = OK 2024-09-14T23:09:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ testlog 'TEST: compare many images' 2024-09-14T23:09:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ tee -a /tmp/tmp.OxLGv2e3xU/rbd-mirror.test.log 2024-09-14T23:09:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ date '+%F %T' 2024-09-14T23:09:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo 2024-09-14 23:09:17 TEST: compare many images 2024-09-14T23:09:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:2024-09-14 23:09:17 TEST: compare many images 2024-09-14T23:09:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 50 2024-09-14T23:09:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:09:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_1 2024-09-14T23:09:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-14T23:09:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:09:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-14T23:09:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-14T23:09:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:09:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-14T23:09:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:09:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:09:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:09:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-14T23:09:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:09:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:09:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:09:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:09:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:09:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:09:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:09:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:09:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:09:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:09:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:09:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-14T23:09:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 2 closest matches: 2024-09-14T23:09:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_11 2024-09-14T23:09:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_19 2024-09-14T23:09:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-14T23:09:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:09:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:09:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:09:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:09:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-14T23:09:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 5 closest matches: 2024-09-14T23:09:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_10 2024-09-14T23:09:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_12 2024-09-14T23:09:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_14 2024-09-14T23:09:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_15 2024-09-14T23:09:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_16 2024-09-14T23:09:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:09:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:09:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:09:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:09:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:09:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-09-14T23:09:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:09:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:09:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:09:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:09:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:09:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:09:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:09:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-14T23:09:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_1.status 2024-09-14T23:09:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_1.status 2024-09-14T23:09:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-14T23:09:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_1.status 2024-09-14T23:09:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.status 2024-09-14T23:09:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-14T23:09:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-14T23:09:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 86 2024-09-14T23:09:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_1.status 2024-09-14T23:09:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_1.status 2024-09-14T23:09:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-14T23:09:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:09:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=86 2024-09-14T23:09:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:09:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.status 2024-09-14T23:09:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.status 2024-09-14T23:09:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-14T23:09:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=35 2024-09-14T23:09:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 86 = 35 2024-09-14T23:09:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:09:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.status 2024-09-14T23:09:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.status 2024-09-14T23:09:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-14T23:09:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=35 2024-09-14T23:09:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 86 = 35 2024-09-14T23:09:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:09:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.status 2024-09-14T23:09:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.status 2024-09-14T23:09:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-14T23:09:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:09:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:09:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:09:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:09:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 86 = '' 2024-09-14T23:09:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:09:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.status 2024-09-14T23:09:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.status 2024-09-14T23:09:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-14T23:09:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=86 2024-09-14T23:09:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 86 = 86 2024-09-14T23:09:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-14T23:09:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:09:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-14T23:09:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_1': 2024-09-14T23:09:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a6f735806f 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a6f735806f 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:00:48 2024 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:00:48 2024 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:00:48 2024 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:09:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:09:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:09:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 23baaf62-5401-42f0-ae98-c17bcc416c67 2024-09-14T23:09:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:09:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-14T23:09:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:09:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:09:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_1.export 2024-09-14T23:09:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.export 2024-09-14T23:09:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_1.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.export 2024-09-14T23:09:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_1.export 2024-09-14T23:09:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.export 2024-09-14T23:09:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:25.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_1.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.export 2024-09-14T23:09:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_1.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_1.export 2024-09-14T23:09:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:09:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_2 2024-09-14T23:09:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-14T23:09:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:09:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-14T23:09:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-14T23:09:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:09:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-14T23:09:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:09:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:09:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:09:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-14T23:09:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:09:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:09:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:09:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:09:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:09:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:09:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:09:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:09:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-14T23:09:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:09:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:09:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:09:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:09:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:09:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-14T23:09:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 2 closest matches: 2024-09-14T23:09:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_23 2024-09-14T23:09:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_27 2024-09-14T23:09:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:09:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:09:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:09:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:09:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-14T23:09:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 2 closest matches: 2024-09-14T23:09:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_20 2024-09-14T23:09:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_28 2024-09-14T23:09:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-14T23:09:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:09:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:09:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:09:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:09:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-14T23:09:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 5 closest matches: 2024-09-14T23:09:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_21 2024-09-14T23:09:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_22 2024-09-14T23:09:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_24 2024-09-14T23:09:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_25 2024-09-14T23:09:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_29 2024-09-14T23:09:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-09-14T23:09:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:09:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:09:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:09:27.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:09:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:09:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:09:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:09:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-09-14T23:09:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:09:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:09:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:09:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:09:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:09:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:09:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:09:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-14T23:09:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_2.status 2024-09-14T23:09:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_2.status 2024-09-14T23:09:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-14T23:09:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_2.status 2024-09-14T23:09:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.status 2024-09-14T23:09:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-14T23:09:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-14T23:09:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 88 2024-09-14T23:09:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_2.status 2024-09-14T23:09:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_2.status 2024-09-14T23:09:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-14T23:09:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:09:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=88 2024-09-14T23:09:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:09:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.status 2024-09-14T23:09:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.status 2024-09-14T23:09:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-14T23:09:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=36 2024-09-14T23:09:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 88 = 36 2024-09-14T23:09:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:09:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.status 2024-09-14T23:09:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.status 2024-09-14T23:09:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-14T23:09:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=36 2024-09-14T23:09:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 88 = 36 2024-09-14T23:09:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:09:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.status 2024-09-14T23:09:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.status 2024-09-14T23:09:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-14T23:09:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:09:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:09:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:09:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:09:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 88 = '' 2024-09-14T23:09:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:09:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.status 2024-09-14T23:09:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.status 2024-09-14T23:09:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-14T23:09:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=88 2024-09-14T23:09:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 88 = 88 2024-09-14T23:09:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-14T23:09:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:09:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-14T23:09:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_2': 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e5dc92ea21 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e5dc92ea21 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:00:58 2024 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:00:58 2024 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:00:58 2024 2024-09-14T23:09:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:09:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:09:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:09:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 13ef0e31-a893-45b8-a2c7-d21116cdc208 2024-09-14T23:09:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:09:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-14T23:09:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:09:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:09:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_2.export 2024-09-14T23:09:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.export 2024-09-14T23:09:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_2.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.export 2024-09-14T23:09:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_2.export 2024-09-14T23:09:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.export 2024-09-14T23:09:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_2.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.export 2024-09-14T23:09:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_2.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_2.export 2024-09-14T23:09:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:09:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_3 2024-09-14T23:09:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-14T23:09:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:09:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-14T23:09:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-14T23:09:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:09:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-14T23:09:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:09:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:09:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:09:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-14T23:09:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:09:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:09:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:09:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:09:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:09:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:09:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:09:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:09:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:09:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:09:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:09:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-14T23:09:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 3 closest matches: 2024-09-14T23:09:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_31 2024-09-14T23:09:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_34 2024-09-14T23:09:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_38 2024-09-14T23:09:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:09:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:09:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:09:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:09:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-14T23:09:35.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 1 closest matches: 2024-09-14T23:09:35.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_39 2024-09-14T23:09:35.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:09:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:09:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:09:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:09:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:09:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-09-14T23:09:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 2 closest matches: 2024-09-14T23:09:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_32 2024-09-14T23:09:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_33 2024-09-14T23:09:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:09:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:09:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:09:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:09:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:09:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:09:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:09:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2024-09-14T23:09:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:09:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:09:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:09:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:09:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:09:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-14T23:09:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_3.status 2024-09-14T23:09:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_3.status 2024-09-14T23:09:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-14T23:09:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_3.status 2024-09-14T23:09:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.status 2024-09-14T23:09:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-14T23:09:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-14T23:09:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 90 2024-09-14T23:09:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_3.status 2024-09-14T23:09:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_3.status 2024-09-14T23:09:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-14T23:09:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:09:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=90 2024-09-14T23:09:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:09:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.status 2024-09-14T23:09:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.status 2024-09-14T23:09:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-14T23:09:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=37 2024-09-14T23:09:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 90 = 37 2024-09-14T23:09:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:09:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.status 2024-09-14T23:09:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.status 2024-09-14T23:09:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-14T23:09:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=37 2024-09-14T23:09:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 90 = 37 2024-09-14T23:09:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:09:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.status 2024-09-14T23:09:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.status 2024-09-14T23:09:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-14T23:09:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:09:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:09:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:09:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:09:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 90 = '' 2024-09-14T23:09:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:09:38.926 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3612237170169814) > weight (0.25) 2024-09-14T23:09:38.926 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8928325453560947) > weight (0.25) 2024-09-14T23:09:38.926 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-14T23:09:38.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-14T23:09:38.927 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T23:09:38.927 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-14T23:09:38.927 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:09:38.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:09:38.926+0000 7f2a5b7e9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 49722) UID: 1000 2024-09-14T23:09:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.status 2024-09-14T23:09:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.status 2024-09-14T23:09:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-14T23:09:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=90 2024-09-14T23:09:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 90 = 90 2024-09-14T23:09:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-14T23:09:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:09:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-14T23:09:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_3': 2024-09-14T23:09:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:09:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:09:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:09:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 120f2c739bde 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.120f2c739bde 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:12 2024 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:12 2024 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:12 2024 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 2903cd12-fe5f-4679-a7de-9ede74f67058 2024-09-14T23:09:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:09:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-14T23:09:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:09:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:09:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_3.export 2024-09-14T23:09:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.export 2024-09-14T23:09:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_3.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.export 2024-09-14T23:09:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_3.export 2024-09-14T23:09:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.export 2024-09-14T23:09:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_3.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.export 2024-09-14T23:09:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_3.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_3.export 2024-09-14T23:09:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:09:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_4 2024-09-14T23:09:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-14T23:09:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:09:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-14T23:09:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:09:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:09:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:09:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:09:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:09:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:09:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:09:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-14T23:09:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:09:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:09:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:09:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:09:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:09:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-14T23:09:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 2 closest matches: 2024-09-14T23:09:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_42 2024-09-14T23:09:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_46 2024-09-14T23:09:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-14T23:09:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:09:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:09:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:09:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:09:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:09:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-14T23:09:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 2 closest matches: 2024-09-14T23:09:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_45 2024-09-14T23:09:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_48 2024-09-14T23:09:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-09-14T23:09:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:09:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:09:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:09:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:09:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:09:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:09:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-09-14T23:09:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:no valid command found; 3 closest matches: 2024-09-14T23:09:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_41 2024-09-14T23:09:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_43 2024-09-14T23:09:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status mirror/image_49 2024-09-14T23:09:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2024-09-14T23:09:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:09:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:09:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:09:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:09:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:09:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:09:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:09:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:09:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:09:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:09:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:09:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:09:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:09:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2024-09-14T23:09:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:09:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:09:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:09:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:09:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:09:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:09:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:09:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-14T23:09:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_4.status 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_4.status 2024-09-14T23:09:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-14T23:09:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_4.status 2024-09-14T23:09:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.status 2024-09-14T23:09:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-14T23:09:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-14T23:09:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 92 2024-09-14T23:09:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_4.status 2024-09-14T23:09:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_4.status 2024-09-14T23:09:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-14T23:09:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:09:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=92 2024-09-14T23:09:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:09:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.status 2024-09-14T23:09:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.status 2024-09-14T23:09:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-14T23:09:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=38 2024-09-14T23:09:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 92 = 38 2024-09-14T23:09:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:09:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.status 2024-09-14T23:09:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.status 2024-09-14T23:09:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-14T23:09:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=38 2024-09-14T23:09:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 92 = 38 2024-09-14T23:09:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:09:45.028 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-14T23:09:45.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T23:09:45.029 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T23:09:45.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T23:09:45.032 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-14T23:09:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.status 2024-09-14T23:09:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.status 2024-09-14T23:09:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-14T23:09:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:45.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:45.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:09:45.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:09:45.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:09:45.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:09:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 92 = '' 2024-09-14T23:09:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:09:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.status 2024-09-14T23:09:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.status 2024-09-14T23:09:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-14T23:09:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=92 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 92 = 92 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:09:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-14T23:09:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_4': 2024-09-14T23:09:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:09:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:09:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:09:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127eafbbb197 2024-09-14T23:09:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127eafbbb197 2024-09-14T23:09:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:09:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:09:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:09:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:09:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:16 2024 2024-09-14T23:09:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:16 2024 2024-09-14T23:09:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:16 2024 2024-09-14T23:09:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:09:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:09:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:09:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: d53f0a91-12f4-4ba8-846e-d09778cdee6d 2024-09-14T23:09:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:09:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-14T23:09:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:09:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:09:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_4.export 2024-09-14T23:09:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.export 2024-09-14T23:09:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_4.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.export 2024-09-14T23:09:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_4.export 2024-09-14T23:09:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.export 2024-09-14T23:09:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_4.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.export 2024-09-14T23:09:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_4.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_4.export 2024-09-14T23:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_5 2024-09-14T23:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-14T23:09:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:09:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-14T23:09:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-14T23:09:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:09:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-14T23:09:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:09:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:09:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:09:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-14T23:09:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:09:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:09:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:09:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:09:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:09:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:09:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:09:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:09:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-14T23:09:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:09:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:09:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:09:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-14T23:09:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:09:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:09:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:09:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:09:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:09:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:09:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:09:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-14T23:09:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_5.status 2024-09-14T23:09:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_5.status 2024-09-14T23:09:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-14T23:09:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_5.status 2024-09-14T23:09:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.status 2024-09-14T23:09:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-14T23:09:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-14T23:09:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 94 2024-09-14T23:09:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_5.status 2024-09-14T23:09:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_5.status 2024-09-14T23:09:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-14T23:09:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:09:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=94 2024-09-14T23:09:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:09:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.status 2024-09-14T23:09:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.status 2024-09-14T23:09:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-14T23:09:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=39 2024-09-14T23:09:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 94 = 39 2024-09-14T23:09:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:09:51.130 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.28391272136583534) > weight (0.25) 2024-09-14T23:09:51.130 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5028296269882859) > weight (0.25) 2024-09-14T23:09:51.130 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9794513155989303) > weight (0.25) 2024-09-14T23:09:51.130 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.26545826182828447) > weight (0.25) 2024-09-14T23:09:51.130 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-09-14T23:09:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.status 2024-09-14T23:09:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.status 2024-09-14T23:09:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-14T23:09:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:09:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:09:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:09:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:09:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 94 = '' 2024-09-14T23:09:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:09:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.status 2024-09-14T23:09:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.status 2024-09-14T23:09:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-14T23:09:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:09:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:09:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:09:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:09:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 94 = '' 2024-09-14T23:09:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:09:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.status 2024-09-14T23:09:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.status 2024-09-14T23:09:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-14T23:09:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=94 2024-09-14T23:09:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 94 = 94 2024-09-14T23:09:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-14T23:09:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:09:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_5': 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a6bcf60a59 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a6bcf60a59 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:09:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:09:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:23 2024 2024-09-14T23:09:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:23 2024 2024-09-14T23:09:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:23 2024 2024-09-14T23:09:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:09:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:09:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:09:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: cd71c6af-1235-405d-96be-3b0fea37da92 2024-09-14T23:09:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:09:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-14T23:09:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:09:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:09:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_5.export 2024-09-14T23:09:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.export 2024-09-14T23:09:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_5.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.export 2024-09-14T23:09:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_5.export 2024-09-14T23:09:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.export 2024-09-14T23:09:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:09:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_5.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.export 2024-09-14T23:09:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_5.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_5.export 2024-09-14T23:09:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:09:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_6 2024-09-14T23:09:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-14T23:09:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:09:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-14T23:09:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:09:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-14T23:09:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:09:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:09:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:09:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:09:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:09:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:09:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:09:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-14T23:09:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-14T23:09:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:09:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:09:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:09:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-14T23:09:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:09:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:09:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:09:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:09:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:09:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:09:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:09:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:09:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:09:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:09:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:09:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:09:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:09:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:09:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:09:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:09:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:09:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:09:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-14T23:09:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:09:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:09:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:09:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:09:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:09:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:09:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:09:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-14T23:09:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_6.status 2024-09-14T23:09:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_6.status 2024-09-14T23:09:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-14T23:09:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_6.status 2024-09-14T23:09:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:09:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.status 2024-09-14T23:09:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-14T23:09:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-14T23:09:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 96 2024-09-14T23:09:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_6.status 2024-09-14T23:09:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:09:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_6.status 2024-09-14T23:09:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-14T23:09:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:09:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=96 2024-09-14T23:09:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:09:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.status 2024-09-14T23:09:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.status 2024-09-14T23:09:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-14T23:09:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:09:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:09:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:09:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:09:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 96 = '' 2024-09-14T23:09:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:09:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.status 2024-09-14T23:09:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.status 2024-09-14T23:09:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-14T23:09:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:09:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:09:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:09:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:09:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:09:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 96 = '' 2024-09-14T23:09:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:09:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:09:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.status 2024-09-14T23:09:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.status 2024-09-14T23:09:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-14T23:09:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:09:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:09:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=96 2024-09-14T23:09:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 96 = 96 2024-09-14T23:09:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:09:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-14T23:09:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:09:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:09:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:09:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:09:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:09:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:09:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-14T23:10:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_6': 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e5cb426385 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e5cb426385 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:33 2024 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:33 2024 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:33 2024 2024-09-14T23:10:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 63095e72-2322-434d-9a80-ed8a5dcddbff 2024-09-14T23:10:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-14T23:10:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:10:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_6.export 2024-09-14T23:10:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.export 2024-09-14T23:10:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_6.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.export 2024-09-14T23:10:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_6.export 2024-09-14T23:10:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.export 2024-09-14T23:10:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:01.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_6.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.export 2024-09-14T23:10:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_6.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_6.export 2024-09-14T23:10:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_7 2024-09-14T23:10:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-14T23:10:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:10:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-14T23:10:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:10:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:10:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:10:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:10:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-14T23:10:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:10:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:10:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:10:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:10:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-14T23:10:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:10:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:10:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:10:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:10:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-14T23:10:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_7.status 2024-09-14T23:10:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_7.status 2024-09-14T23:10:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-14T23:10:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_7.status 2024-09-14T23:10:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.status 2024-09-14T23:10:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-14T23:10:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-14T23:10:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 98 2024-09-14T23:10:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_7.status 2024-09-14T23:10:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_7.status 2024-09-14T23:10:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-14T23:10:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:10:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=98 2024-09-14T23:10:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:10:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.status 2024-09-14T23:10:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.status 2024-09-14T23:10:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-14T23:10:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 98 = '' 2024-09-14T23:10:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:10:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.status 2024-09-14T23:10:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.status 2024-09-14T23:10:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-14T23:10:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 98 = '' 2024-09-14T23:10:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:10:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.status 2024-09-14T23:10:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.status 2024-09-14T23:10:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-14T23:10:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=98 2024-09-14T23:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 98 = 98 2024-09-14T23:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-14T23:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_7': 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 120f7f0d3e05 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.120f7f0d3e05 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:45 2024 2024-09-14T23:10:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:45 2024 2024-09-14T23:10:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:45 2024 2024-09-14T23:10:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: b412e86d-c0d6-4c66-9336-1d3894abecba 2024-09-14T23:10:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-14T23:10:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:10:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_7.export 2024-09-14T23:10:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.export 2024-09-14T23:10:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_7.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.export 2024-09-14T23:10:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_7.export 2024-09-14T23:10:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.export 2024-09-14T23:10:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_7.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.export 2024-09-14T23:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_7.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_7.export 2024-09-14T23:10:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_8 2024-09-14T23:10:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-14T23:10:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:10:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-14T23:10:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-14T23:10:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:10:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-14T23:10:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:10:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:10:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:10:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-14T23:10:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:10:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:10:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:10:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:10:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:10:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:10:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:10:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:10:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:10:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:10:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-14T23:10:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-14T23:10:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:10:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:10:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:10:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:10:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-14T23:10:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:10:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:10:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:10:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:10:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-14T23:10:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-14T23:10:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_8.status 2024-09-14T23:10:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_8.status 2024-09-14T23:10:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-14T23:10:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_8.status 2024-09-14T23:10:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.status 2024-09-14T23:10:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-14T23:10:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-14T23:10:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 100 2024-09-14T23:10:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_8.status 2024-09-14T23:10:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_8.status 2024-09-14T23:10:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-14T23:10:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:10.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:10:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=100 2024-09-14T23:10:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:10:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.status 2024-09-14T23:10:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.status 2024-09-14T23:10:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-14T23:10:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 100 = '' 2024-09-14T23:10:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:10:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.status 2024-09-14T23:10:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.status 2024-09-14T23:10:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-14T23:10:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 100 = '' 2024-09-14T23:10:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:10:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.status 2024-09-14T23:10:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.status 2024-09-14T23:10:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-14T23:10:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=100 2024-09-14T23:10:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 100 = 100 2024-09-14T23:10:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-14T23:10:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:10:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-14T23:10:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_8': 2024-09-14T23:10:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127e49e44317 2024-09-14T23:10:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127e49e44317 2024-09-14T23:10:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:01:54 2024 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:01:54 2024 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:01:54 2024 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 1183b733-6e8d-4fa0-bcc5-8cdba88bdd37 2024-09-14T23:10:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-14T23:10:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:10:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:13.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_8.export 2024-09-14T23:10:13.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.export 2024-09-14T23:10:13.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_8.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.export 2024-09-14T23:10:13.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_8.export 2024-09-14T23:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.export 2024-09-14T23:10:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_8.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.export 2024-09-14T23:10:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_8.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_8.export 2024-09-14T23:10:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_9 2024-09-14T23:10:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-14T23:10:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:10:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-14T23:10:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-14T23:10:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:10:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-14T23:10:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:10:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:10:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:10:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-14T23:10:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:10:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:10:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:10:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:10:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:10:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:10:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:10:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:10:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:10:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:10:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-14T23:10:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:10:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:10:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:10:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:10:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:10:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:10:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:10:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-14T23:10:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-14T23:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-14T23:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_9.status 2024-09-14T23:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_9.status 2024-09-14T23:10:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-14T23:10:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_9.status 2024-09-14T23:10:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.status 2024-09-14T23:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-14T23:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-14T23:10:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 102 2024-09-14T23:10:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_9.status 2024-09-14T23:10:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_9.status 2024-09-14T23:10:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-14T23:10:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:10:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=102 2024-09-14T23:10:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:10:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.status 2024-09-14T23:10:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.status 2024-09-14T23:10:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-14T23:10:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 102 = '' 2024-09-14T23:10:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:10:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.status 2024-09-14T23:10:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.status 2024-09-14T23:10:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-14T23:10:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 102 = '' 2024-09-14T23:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:10:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.status 2024-09-14T23:10:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.status 2024-09-14T23:10:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-14T23:10:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=102 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 102 = 102 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:10:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_9': 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10a6649f9950 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10a6649f9950 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:02:05 2024 2024-09-14T23:10:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:02:05 2024 2024-09-14T23:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:02:05 2024 2024-09-14T23:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: e951d895-861e-422c-881e-e2fec0aaeafd 2024-09-14T23:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-14T23:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_9.export 2024-09-14T23:10:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.export 2024-09-14T23:10:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_9.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.export 2024-09-14T23:10:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_9.export 2024-09-14T23:10:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.export 2024-09-14T23:10:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_9.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.export 2024-09-14T23:10:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_9.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_9.export 2024-09-14T23:10:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_10 2024-09-14T23:10:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-14T23:10:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-14T23:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:10:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-14T23:10:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:10:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:10:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:10:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:10:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:10:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:10:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:10:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-14T23:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:10:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:10:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:10:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-14T23:10:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-14T23:10:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:10:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:10:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-14T23:10:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:10:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:10:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:10:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:10:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:10:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:10:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-14T23:10:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:10:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:10:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:10:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:10:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:10:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:10:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:10:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-14T23:10:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:10:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:10:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-14T23:10:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-14T23:10:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_10.status 2024-09-14T23:10:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_10.status 2024-09-14T23:10:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-14T23:10:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_10.status 2024-09-14T23:10:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.status 2024-09-14T23:10:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-14T23:10:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-14T23:10:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 104 2024-09-14T23:10:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_10.status 2024-09-14T23:10:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_10.status 2024-09-14T23:10:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-14T23:10:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:10:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=104 2024-09-14T23:10:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:10:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.status 2024-09-14T23:10:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.status 2024-09-14T23:10:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-14T23:10:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 104 = '' 2024-09-14T23:10:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:10:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.status 2024-09-14T23:10:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.status 2024-09-14T23:10:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-14T23:10:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 104 = '' 2024-09-14T23:10:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.status 2024-09-14T23:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.status 2024-09-14T23:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-14T23:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=104 2024-09-14T23:10:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 104 = 104 2024-09-14T23:10:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-14T23:10:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:10:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_10': 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e56f672ee0 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e56f672ee0 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:02:16 2024 2024-09-14T23:10:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:02:16 2024 2024-09-14T23:10:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:02:16 2024 2024-09-14T23:10:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: fd912f05-09a2-4d03-a711-51d6bfba6b89 2024-09-14T23:10:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-14T23:10:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:10:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_10.export 2024-09-14T23:10:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.export 2024-09-14T23:10:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_10.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.export 2024-09-14T23:10:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_10.export 2024-09-14T23:10:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.export 2024-09-14T23:10:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_10.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.export 2024-09-14T23:10:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_10.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_10.export 2024-09-14T23:10:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_11 2024-09-14T23:10:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-14T23:10:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:26.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:10:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:10:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:10:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:10:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:10:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:10:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:10:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:10:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:10:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:10:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:10:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:10:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-14T23:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:10:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:10:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:10:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:10:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:10:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:10:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:10:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-14T23:10:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_11.status 2024-09-14T23:10:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_11.status 2024-09-14T23:10:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-14T23:10:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_11.status 2024-09-14T23:10:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.status 2024-09-14T23:10:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-14T23:10:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-14T23:10:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 106 2024-09-14T23:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_11.status 2024-09-14T23:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_11.status 2024-09-14T23:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-14T23:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:10:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=106 2024-09-14T23:10:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:10:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.status 2024-09-14T23:10:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.status 2024-09-14T23:10:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-14T23:10:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 106 = '' 2024-09-14T23:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.status 2024-09-14T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.status 2024-09-14T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-14T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 106 = '' 2024-09-14T23:10:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:10:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.status 2024-09-14T23:10:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.status 2024-09-14T23:10:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-14T23:10:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=106 2024-09-14T23:10:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 106 = 106 2024-09-14T23:10:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-14T23:10:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:10:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-14T23:10:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_11': 2024-09-14T23:10:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 120fcca9b222 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.120fcca9b222 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:02:26 2024 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:02:26 2024 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:02:26 2024 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 3d92657a-d871-4f0e-87e5-49062e513296 2024-09-14T23:10:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-14T23:10:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:10:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_11.export 2024-09-14T23:10:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.export 2024-09-14T23:10:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_11.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.export 2024-09-14T23:10:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_11.export 2024-09-14T23:10:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.export 2024-09-14T23:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_11.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.export 2024-09-14T23:10:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_11.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_11.export 2024-09-14T23:10:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_12 2024-09-14T23:10:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-14T23:10:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-14T23:10:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:10:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:10:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:10:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-14T23:10:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:10:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:10:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:10:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:10:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:10:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:10:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:10:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:10:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-14T23:10:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:10:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:10:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:10:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:10:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:10:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:10:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:10:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:10:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-14T23:10:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-14T23:10:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:10:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:10:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-14T23:10:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:10:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:10:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:10:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:10:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:10:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:10:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:10:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:10:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:10:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:10:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:10:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:10:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-14T23:10:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:10:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:10:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:10:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:10:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:10:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:10:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:10:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-14T23:10:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_12.status 2024-09-14T23:10:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_12.status 2024-09-14T23:10:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-14T23:10:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_12.status 2024-09-14T23:10:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.status 2024-09-14T23:10:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-14T23:10:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-14T23:10:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 108 2024-09-14T23:10:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_12.status 2024-09-14T23:10:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_12.status 2024-09-14T23:10:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-14T23:10:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:10:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=108 2024-09-14T23:10:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:10:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.status 2024-09-14T23:10:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.status 2024-09-14T23:10:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-14T23:10:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 108 = '' 2024-09-14T23:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:10:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.status 2024-09-14T23:10:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.status 2024-09-14T23:10:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-14T23:10:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 108 = '' 2024-09-14T23:10:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:10:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.status 2024-09-14T23:10:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.status 2024-09-14T23:10:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-14T23:10:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=108 2024-09-14T23:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 108 = 108 2024-09-14T23:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-14T23:10:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:10:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-14T23:10:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_12': 2024-09-14T23:10:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127e3d4e2e64 2024-09-14T23:10:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127e3d4e2e64 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:02:40 2024 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:02:40 2024 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:02:40 2024 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 64ee2f3d-5dac-4606-aaf0-b005a4416265 2024-09-14T23:10:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-14T23:10:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:10:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_12.export 2024-09-14T23:10:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.export 2024-09-14T23:10:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_12.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.export 2024-09-14T23:10:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_12.export 2024-09-14T23:10:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.export 2024-09-14T23:10:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_12.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.export 2024-09-14T23:10:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_12.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_12.export 2024-09-14T23:10:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_13 2024-09-14T23:10:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-14T23:10:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:10:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-14T23:10:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-14T23:10:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:10:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-14T23:10:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:10:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:10:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:10:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-14T23:10:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:10:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:10:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:10:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:10:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:10:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:10:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:10:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:10:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:10:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:10:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:10:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:10:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:10:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:10:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:10:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:10:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-14T23:10:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-14T23:10:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:10:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:10:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:10:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:10:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-09-14T23:10:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-14T23:10:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:10:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:10:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:10:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:10:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:10:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:10:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:10:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:10:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:10:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:10:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2024-09-14T23:10:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-14T23:10:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:10:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:10:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:10:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:10:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:10:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:10:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:10:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:10:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:10:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2024-09-14T23:10:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:10:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:10:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:10:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:10:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:10:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:10:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:10:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-14T23:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_13.status 2024-09-14T23:10:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_13.status 2024-09-14T23:10:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-14T23:10:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_13.status 2024-09-14T23:10:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.status 2024-09-14T23:10:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-14T23:10:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-14T23:10:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 110 2024-09-14T23:10:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_13.status 2024-09-14T23:10:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_13.status 2024-09-14T23:10:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-14T23:10:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:10:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=110 2024-09-14T23:10:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:10:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.status 2024-09-14T23:10:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.status 2024-09-14T23:10:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-14T23:10:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=47 2024-09-14T23:10:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 110 = 47 2024-09-14T23:10:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:10:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.status 2024-09-14T23:10:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.status 2024-09-14T23:10:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-14T23:10:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=47 2024-09-14T23:10:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 110 = 47 2024-09-14T23:10:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:10:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.status 2024-09-14T23:10:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.status 2024-09-14T23:10:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-14T23:10:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 110 = '' 2024-09-14T23:10:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:10:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.status 2024-09-14T23:10:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.status 2024-09-14T23:10:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-14T23:10:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=110 2024-09-14T23:10:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 110 = 110 2024-09-14T23:10:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-14T23:10:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:10:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_13': 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 120f25479678 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.120f25479678 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:24 2024 2024-09-14T23:10:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:24 2024 2024-09-14T23:10:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:24 2024 2024-09-14T23:10:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: ca7825af-19b6-473a-a8e5-328e157a8d1a 2024-09-14T23:10:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-14T23:10:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:10:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_13.export 2024-09-14T23:10:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.export 2024-09-14T23:10:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_13.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.export 2024-09-14T23:10:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_13.export 2024-09-14T23:10:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.export 2024-09-14T23:10:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_13.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.export 2024-09-14T23:10:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_13.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_13.export 2024-09-14T23:10:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_14 2024-09-14T23:10:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-14T23:10:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:10:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-14T23:10:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:10:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:10:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:10:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-14T23:10:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:10:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:10:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:10:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:10:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:10:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:10:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:10:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:10:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-14T23:10:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:10:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:10:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:10:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:10:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-14T23:10:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-14T23:10:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:10:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:10:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:10:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:10:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-14T23:10:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:10:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:10:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:10:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:10:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:10:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:10:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:10:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-14T23:10:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_14.status 2024-09-14T23:10:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_14.status 2024-09-14T23:10:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-14T23:10:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_14.status 2024-09-14T23:10:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.status 2024-09-14T23:10:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-14T23:10:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-14T23:10:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 112 2024-09-14T23:10:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_14.status 2024-09-14T23:10:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_14.status 2024-09-14T23:10:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-14T23:10:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:10:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=112 2024-09-14T23:10:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:10:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.status 2024-09-14T23:10:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.status 2024-09-14T23:10:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-14T23:10:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=48 2024-09-14T23:10:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 112 = 48 2024-09-14T23:10:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:10:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.status 2024-09-14T23:10:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.status 2024-09-14T23:10:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-14T23:10:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=48 2024-09-14T23:10:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 112 = 48 2024-09-14T23:10:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:10:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.status 2024-09-14T23:10:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.status 2024-09-14T23:10:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-14T23:10:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 112 = '' 2024-09-14T23:10:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:10:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.status 2024-09-14T23:10:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.status 2024-09-14T23:10:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-14T23:10:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=112 2024-09-14T23:10:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 112 = 112 2024-09-14T23:10:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-14T23:10:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:10:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-14T23:10:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_14': 2024-09-14T23:10:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 128c1b255168 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.128c1b255168 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:04 2024 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:04 2024 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:04 2024 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 45c698d2-2f4b-4209-84b4-d26cd2e23680 2024-09-14T23:10:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-14T23:10:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:10:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_14.export 2024-09-14T23:10:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.export 2024-09-14T23:10:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_14.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.export 2024-09-14T23:10:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_14.export 2024-09-14T23:10:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:51.879 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.export 2024-09-14T23:10:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:52.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_14.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.export 2024-09-14T23:10:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_14.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_14.export 2024-09-14T23:10:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_15 2024-09-14T23:10:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-14T23:10:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:10:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-14T23:10:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-14T23:10:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:10:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:53.034 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4078169319445346) > weight (0.25) 2024-09-14T23:10:53.034 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6524518596470777) > weight (0.25) 2024-09-14T23:10:53.034 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.441306108403831) > weight (0.25) 2024-09-14T23:10:53.034 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-14T23:10:53.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-14T23:10:53.034 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T23:10:53.035 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-14T23:10:53.035 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:10:53.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:10:53.034+0000 7fc8ea610640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 48400) UID: 1000 2024-09-14T23:10:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-14T23:10:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:10:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:10:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:10:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:10:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:10:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:10:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-14T23:10:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-14T23:10:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:10:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:10:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:10:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:10:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:10:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:10:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:10:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-09-14T23:10:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:10:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:10:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:10:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:10:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:10:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:10:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:10:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-14T23:10:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:10:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:10:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-14T23:10:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:10:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-14T23:10:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_15.status 2024-09-14T23:10:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_15.status 2024-09-14T23:10:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-14T23:10:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_15.status 2024-09-14T23:10:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:10:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.status 2024-09-14T23:10:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-14T23:10:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-14T23:10:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 114 2024-09-14T23:10:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_15.status 2024-09-14T23:10:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_15.status 2024-09-14T23:10:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-14T23:10:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:10:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=114 2024-09-14T23:10:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:10:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.status 2024-09-14T23:10:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.status 2024-09-14T23:10:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-14T23:10:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=49 2024-09-14T23:10:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 114 = 49 2024-09-14T23:10:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:10:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.status 2024-09-14T23:10:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.status 2024-09-14T23:10:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-14T23:10:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=49 2024-09-14T23:10:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 114 = 49 2024-09-14T23:10:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:10:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.status 2024-09-14T23:10:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.status 2024-09-14T23:10:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-14T23:10:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:10:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:10:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:10:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:10:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:10:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 114 = '' 2024-09-14T23:10:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:10:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:10:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.status 2024-09-14T23:10:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.status 2024-09-14T23:10:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-14T23:10:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:10:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:10:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=114 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 114 = 114 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:10:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:10:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_15': 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 128ccabfab24 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.128ccabfab24 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:11 2024 2024-09-14T23:10:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:11 2024 2024-09-14T23:10:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:11 2024 2024-09-14T23:10:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:10:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:10:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:10:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 77c7ed00-9d21-46b0-b7ed-332bbd914227 2024-09-14T23:10:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:10:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-14T23:10:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:10:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:10:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_15.export 2024-09-14T23:10:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.export 2024-09-14T23:10:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_15.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.export 2024-09-14T23:10:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_15.export 2024-09-14T23:10:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.export 2024-09-14T23:10:59.136 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-14T23:10:59.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-14T23:10:59.137 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-14T23:10:59.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-14T23:10:59.139 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-14T23:10:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:10:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_15.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.export 2024-09-14T23:10:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_15.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_15.export 2024-09-14T23:10:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:10:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:10:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_16 2024-09-14T23:10:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-14T23:10:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:10:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:10:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:10:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:10:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:11:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:11:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:11:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-14T23:11:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:11:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:11:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:11:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-14T23:11:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:11:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:11:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:11:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:11:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:11:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:11:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:11:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:11:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:11:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:11:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-14T23:11:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:11:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:11:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:11:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:11:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:11:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:11:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:11:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:11:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:11:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:11:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:11:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:11:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:11:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:11:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:11:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-14T23:11:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-14T23:11:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:11:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:11:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:11:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:11:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:11:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:11:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:11:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:11:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:11:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-14T23:11:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:11:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_16.status 2024-09-14T23:11:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:11:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_16.status 2024-09-14T23:11:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-14T23:11:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_16.status 2024-09-14T23:11:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.status 2024-09-14T23:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-14T23:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-14T23:11:02.424 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 116 2024-09-14T23:11:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_16.status 2024-09-14T23:11:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_16.status 2024-09-14T23:11:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-14T23:11:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:11:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=116 2024-09-14T23:11:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:11:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.status 2024-09-14T23:11:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.status 2024-09-14T23:11:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-14T23:11:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=50 2024-09-14T23:11:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 116 = 50 2024-09-14T23:11:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:11:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.status 2024-09-14T23:11:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.status 2024-09-14T23:11:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-14T23:11:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=50 2024-09-14T23:11:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 116 = 50 2024-09-14T23:11:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:11:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.status 2024-09-14T23:11:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.status 2024-09-14T23:11:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-14T23:11:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 116 = '' 2024-09-14T23:11:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:11:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.status 2024-09-14T23:11:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.status 2024-09-14T23:11:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-14T23:11:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=116 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 116 = 116 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:11:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:11:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-14T23:11:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_16': 2024-09-14T23:11:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:11:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:11:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:11:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 128c54662f18 2024-09-14T23:11:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.128c54662f18 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:20 2024 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:20 2024 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:20 2024 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:11:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: ff85a14f-4301-4076-86de-a3f6e6b1ddd9 2024-09-14T23:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:11:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-14T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_16.export 2024-09-14T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.export 2024-09-14T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_16.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.export 2024-09-14T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_16.export 2024-09-14T23:11:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:11:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.export 2024-09-14T23:11:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:11:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_16.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.export 2024-09-14T23:11:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_16.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_16.export 2024-09-14T23:11:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:11:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_17 2024-09-14T23:11:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-14T23:11:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:11:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:11:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-14T23:11:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-14T23:11:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:11:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:11:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:11:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:11:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-14T23:11:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:11:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:11:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:11:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-14T23:11:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:11:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:11:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:11:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:11:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:11:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:11:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:11:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:11:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:11:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:11:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-14T23:11:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:11:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:11:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:11:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:11:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:11:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:11:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:11:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:11:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:11:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:11:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:11:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:11:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:11:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:11:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:11:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-14T23:11:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-14T23:11:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:11:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:11:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:11:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-09-14T23:11:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:11:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:11:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:11:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:11:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:11:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:11:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:11:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:11:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:11:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:11:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-09-14T23:11:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-14T23:11:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:11:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:11:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:11:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:11:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:11:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:11:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:11:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:11:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:11:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:11:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:11:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:11:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-09-14T23:11:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:11:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:11:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:11:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:11:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:11:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-14T23:11:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:11:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-14T23:11:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_17.status 2024-09-14T23:11:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:11:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_17.status 2024-09-14T23:11:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-14T23:11:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_17.status 2024-09-14T23:11:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:11:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.status 2024-09-14T23:11:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-14T23:11:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-14T23:11:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 118 2024-09-14T23:11:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_17.status 2024-09-14T23:11:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_17.status 2024-09-14T23:11:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-14T23:11:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:11:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=118 2024-09-14T23:11:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:11:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.status 2024-09-14T23:11:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.status 2024-09-14T23:11:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T23:11:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=51 2024-09-14T23:11:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 118 = 51 2024-09-14T23:11:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:11:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.status 2024-09-14T23:11:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.status 2024-09-14T23:11:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T23:11:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=51 2024-09-14T23:11:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 118 = 51 2024-09-14T23:11:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:11:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.status 2024-09-14T23:11:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.status 2024-09-14T23:11:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T23:11:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 118 = '' 2024-09-14T23:11:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:11:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.status 2024-09-14T23:11:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.status 2024-09-14T23:11:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T23:11:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=118 2024-09-14T23:11:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 118 = 118 2024-09-14T23:11:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-14T23:11:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:11:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:11:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:11:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:11:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-14T23:11:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_17': 2024-09-14T23:11:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:11:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:11:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:11:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e573ed442a 2024-09-14T23:11:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e573ed442a 2024-09-14T23:11:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:31 2024 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:31 2024 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:31 2024 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 839cf198-206a-4346-91c9-f4360063f32f 2024-09-14T23:11:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:11:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-14T23:11:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:11:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:11:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:11:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_17.export 2024-09-14T23:11:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.export 2024-09-14T23:11:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_17.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.export 2024-09-14T23:11:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_17.export 2024-09-14T23:11:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:11:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.export 2024-09-14T23:11:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:11:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_17.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.export 2024-09-14T23:11:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_17.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_17.export 2024-09-14T23:11:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:11:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_18 2024-09-14T23:11:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-14T23:11:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:11:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:11:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-14T23:11:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-14T23:11:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:11:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:11:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:11:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:11:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-14T23:11:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:11:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:11:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:11:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-14T23:11:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-14T23:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:11:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:11:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:11:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:11:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:11:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:11:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:11:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:11:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:11:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-14T23:11:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-14T23:11:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:11:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:11:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:11:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-14T23:11:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:11:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:11:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:11:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:11:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:11:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:11:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:11:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:11:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:11:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:11:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:11:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:11:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:11:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:11:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:11:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:11:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:11:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-14T23:11:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-14T23:11:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:11:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:11:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:11:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:11:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:11:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:11:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:11:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:11:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:11:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:11:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-09-14T23:11:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:11:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:11:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:11:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:11:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:11:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:11:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:11:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-14T23:11:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:11:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_18.status 2024-09-14T23:11:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:11:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_18.status 2024-09-14T23:11:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-14T23:11:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_18.status 2024-09-14T23:11:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:11:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.status 2024-09-14T23:11:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-14T23:11:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-14T23:11:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 120 2024-09-14T23:11:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_18.status 2024-09-14T23:11:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_18.status 2024-09-14T23:11:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-14T23:11:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:11:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=120 2024-09-14T23:11:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:11:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.status 2024-09-14T23:11:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.status 2024-09-14T23:11:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-14T23:11:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=52 2024-09-14T23:11:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 120 = 52 2024-09-14T23:11:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:11:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.status 2024-09-14T23:11:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.status 2024-09-14T23:11:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-14T23:11:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=52 2024-09-14T23:11:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 120 = 52 2024-09-14T23:11:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:11:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.status 2024-09-14T23:11:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.status 2024-09-14T23:11:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-14T23:11:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 120 = '' 2024-09-14T23:11:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:11:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.status 2024-09-14T23:11:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.status 2024-09-14T23:11:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-14T23:11:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=120 2024-09-14T23:11:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 120 = 120 2024-09-14T23:11:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-14T23:11:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:11:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:11:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:11:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:11:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-14T23:11:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_18': 2024-09-14T23:11:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:11:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:11:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:11:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 120f7b4b6075 2024-09-14T23:11:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.120f7b4b6075 2024-09-14T23:11:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:11:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:38 2024 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:38 2024 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:38 2024 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: d80a064e-80a3-476f-9ce2-c2191cc27f0b 2024-09-14T23:11:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:11:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-14T23:11:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:11:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:11:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:11:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_18.export 2024-09-14T23:11:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.export 2024-09-14T23:11:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_18.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.export 2024-09-14T23:11:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_18.export 2024-09-14T23:11:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:11:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.export 2024-09-14T23:11:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:11:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_18.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.export 2024-09-14T23:11:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_18.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_18.export 2024-09-14T23:11:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:11:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_19 2024-09-14T23:11:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-14T23:11:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:11:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:11:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-14T23:11:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-14T23:11:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:11:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:11:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:11:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:11:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-14T23:11:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:11:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:11:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:11:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-14T23:11:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:11:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:11:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:11:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:11:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:11:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:11:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:11:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:11:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:11:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:11:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:11:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:11:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:11:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:11:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:11:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:11:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-14T23:11:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:11:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:11:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:11:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:11:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:11:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-14T23:11:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_19.status 2024-09-14T23:11:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:11:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_19.status 2024-09-14T23:11:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-14T23:11:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_19.status 2024-09-14T23:11:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:11:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-14T23:11:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-14T23:11:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 122 2024-09-14T23:11:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_19.status 2024-09-14T23:11:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:11:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_19.status 2024-09-14T23:11:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:11:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=122 2024-09-14T23:11:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:11:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=53 2024-09-14T23:11:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = 53 2024-09-14T23:11:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:11:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=53 2024-09-14T23:11:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = 53 2024-09-14T23:11:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:11:25.131 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T23:11:25.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T23:11:25.131 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-14T23:11:25.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:11:25.130+0000 7fdfc4e04640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 45386) UID: 1000 2024-09-14T23:11:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = '' 2024-09-14T23:11:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:11:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:11:26.501+0000 7f39be35e640 -1 rbd::mirror::ImageReplayer: 0x55d65b0ee780 [3/fd912f05-09a2-4d03-a711-51d6bfba6b89] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:11:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:11:26.501+0000 7f39beb5f640 -1 rbd::mirror::ImageReplayer: 0x55d65b0ee780 [3/fd912f05-09a2-4d03-a711-51d6bfba6b89] register_admin_socket_hook: error registering admin socket commands 2024-09-14T23:11:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = '' 2024-09-14T23:11:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = '' 2024-09-14T23:11:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:11:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = '' 2024-09-14T23:11:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:11:33.132 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T23:11:33.133 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:11:33.133 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T23:11:33.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T23:11:33.133 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T23:11:33.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T23:11:33.136 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-14T23:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = '' 2024-09-14T23:11:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:11:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = '' 2024-09-14T23:11:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:11:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = '' 2024-09-14T23:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:11:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:11:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:11:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:11:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:11:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:11:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:11:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:11:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:11:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:11:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:11:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:11:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:11:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = '' 2024-09-14T23:11:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:11:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:11:58.563 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T23:11:58.561+0000 7f5fe0b61640 -1 reset not still connected to 0x559281416820 2024-09-14T23:11:58.563 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T23:11:58.561+0000 7f5fe0b61640 -1 reset not still connected to 0x559281928d00 2024-09-14T23:12:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:12:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:12:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.status 2024-09-14T23:12:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T23:12:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=122 2024-09-14T23:12:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 122 = 122 2024-09-14T23:12:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-14T23:12:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:12:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:12:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-14T23:12:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_19': 2024-09-14T23:12:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:12:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:12:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:12:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127e64841943 2024-09-14T23:12:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127e64841943 2024-09-14T23:12:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:12:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:12:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:12:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:12:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:51 2024 2024-09-14T23:12:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:51 2024 2024-09-14T23:12:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:51 2024 2024-09-14T23:12:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:12:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:12:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:12:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5b5b353a-9976-4517-9150-396dd4235407 2024-09-14T23:12:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:12:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-14T23:12:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:12:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:12:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_19.export 2024-09-14T23:12:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.export 2024-09-14T23:12:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_19.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.export 2024-09-14T23:12:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_19.export 2024-09-14T23:12:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.export 2024-09-14T23:12:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_19.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.export 2024-09-14T23:12:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_19.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_19.export 2024-09-14T23:12:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:12:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_20 2024-09-14T23:12:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-14T23:12:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:12:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-14T23:12:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-14T23:12:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:12:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-14T23:12:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:12:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:12:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:12:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-14T23:12:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:12:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:12:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:12:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:12:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:12:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:12:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:12:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:12:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-14T23:12:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:12:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:12:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:12:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:12:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:12:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:12:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:12:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-14T23:12:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-14T23:12:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-14T23:12:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:12:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:12:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:12:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:12:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-14T23:12:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:12:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:12:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:12:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:12:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:12:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:12:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:12:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-14T23:12:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_20.status 2024-09-14T23:12:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_20.status 2024-09-14T23:12:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-14T23:12:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_20.status 2024-09-14T23:12:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.status 2024-09-14T23:12:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-14T23:12:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-14T23:12:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 124 2024-09-14T23:12:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_20.status 2024-09-14T23:12:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_20.status 2024-09-14T23:12:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-14T23:12:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:12:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=124 2024-09-14T23:12:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:12:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.status 2024-09-14T23:12:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.status 2024-09-14T23:12:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-14T23:12:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=54 2024-09-14T23:12:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 124 = 54 2024-09-14T23:12:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:12:16.140 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5986741507231408) > weight (0.25) 2024-09-14T23:12:16.140 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-14T23:12:16.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-14T23:12:16.140 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-14T23:12:16.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:12:16.140+0000 7f7ab02c1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 43583) UID: 1000 2024-09-14T23:12:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.status 2024-09-14T23:12:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.status 2024-09-14T23:12:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-14T23:12:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 124 = '' 2024-09-14T23:12:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:12:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.status 2024-09-14T23:12:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.status 2024-09-14T23:12:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-14T23:12:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 124 = '' 2024-09-14T23:12:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:12:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.status 2024-09-14T23:12:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.status 2024-09-14T23:12:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-14T23:12:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=124 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 124 = 124 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:12:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-14T23:12:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_20': 2024-09-14T23:12:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:12:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 128ce088502c 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.128ce088502c 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:03:59 2024 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:03:59 2024 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:03:59 2024 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: c471720c-4eb1-4c8a-a244-9687727d167b 2024-09-14T23:12:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:12:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-14T23:12:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:12:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:12:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_20.export 2024-09-14T23:12:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.export 2024-09-14T23:12:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_20.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.export 2024-09-14T23:12:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_20.export 2024-09-14T23:12:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.export 2024-09-14T23:12:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_20.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.export 2024-09-14T23:12:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_20.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_20.export 2024-09-14T23:12:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:12:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_21 2024-09-14T23:12:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-14T23:12:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:12:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-14T23:12:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-14T23:12:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:12:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-14T23:12:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:12:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:12:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:12:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:12:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:12:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:12:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:12:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:12:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:12:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:12:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:12:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:12:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:12:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-14T23:12:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-14T23:12:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-09-14T23:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:12:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:12:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:12:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-09-14T23:12:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-14T23:12:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:12:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:12:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2024-09-14T23:12:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:12:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:12:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:12:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:12:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:12:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-14T23:12:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_21.status 2024-09-14T23:12:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_21.status 2024-09-14T23:12:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-14T23:12:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_21.status 2024-09-14T23:12:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.status 2024-09-14T23:12:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-14T23:12:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-14T23:12:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 126 2024-09-14T23:12:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_21.status 2024-09-14T23:12:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_21.status 2024-09-14T23:12:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-14T23:12:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:12:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=126 2024-09-14T23:12:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:12:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.status 2024-09-14T23:12:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.status 2024-09-14T23:12:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-14T23:12:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=55 2024-09-14T23:12:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 126 = 55 2024-09-14T23:12:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:12:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.status 2024-09-14T23:12:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.status 2024-09-14T23:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-14T23:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 126 = '' 2024-09-14T23:12:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:12:25.141 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-14T23:12:25.141 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:12:25.141 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-14T23:12:25.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T23:12:25.142 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T23:12:25.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T23:12:25.145 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-14T23:12:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.status 2024-09-14T23:12:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.status 2024-09-14T23:12:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-14T23:12:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 126 = '' 2024-09-14T23:12:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:12:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.status 2024-09-14T23:12:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.status 2024-09-14T23:12:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-14T23:12:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=126 2024-09-14T23:12:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 126 = 126 2024-09-14T23:12:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-14T23:12:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:12:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-14T23:12:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_21': 2024-09-14T23:12:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:12:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:12:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:12:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e52aadf0a5 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e52aadf0a5 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:11 2024 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:11 2024 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:11 2024 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 67fd617c-03d8-493d-bbbb-1c4c9b42e7b8 2024-09-14T23:12:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:12:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-14T23:12:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:12:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:12:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_21.export 2024-09-14T23:12:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.export 2024-09-14T23:12:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_21.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.export 2024-09-14T23:12:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_21.export 2024-09-14T23:12:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.export 2024-09-14T23:12:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_21.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.export 2024-09-14T23:12:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_21.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_21.export 2024-09-14T23:12:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:12:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_22 2024-09-14T23:12:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-14T23:12:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:12:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-14T23:12:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-14T23:12:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:12:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-14T23:12:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:12:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:12:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:12:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-14T23:12:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:12:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:12:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:12:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:12:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:12:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:12:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:12:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:12:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-14T23:12:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:12:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:12:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:12:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:12:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:12:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-14T23:12:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-14T23:12:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-14T23:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:12:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-14T23:12:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-14T23:12:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-09-14T23:12:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:12:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:12:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:12:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-09-14T23:12:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:12:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:12:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:12:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:12:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:12:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:12:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:12:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-14T23:12:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_22.status 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_22.status 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-14T23:12:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_22.status 2024-09-14T23:12:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.status 2024-09-14T23:12:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-14T23:12:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-14T23:12:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 128 2024-09-14T23:12:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_22.status 2024-09-14T23:12:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_22.status 2024-09-14T23:12:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-14T23:12:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=128 2024-09-14T23:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:12:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.status 2024-09-14T23:12:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.status 2024-09-14T23:12:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-14T23:12:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=56 2024-09-14T23:12:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 128 = 56 2024-09-14T23:12:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:12:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.status 2024-09-14T23:12:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.status 2024-09-14T23:12:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-14T23:12:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 128 = '' 2024-09-14T23:12:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:12:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.status 2024-09-14T23:12:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.status 2024-09-14T23:12:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-14T23:12:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 128 = '' 2024-09-14T23:12:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:12:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.status 2024-09-14T23:12:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.status 2024-09-14T23:12:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-14T23:12:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=128 2024-09-14T23:12:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 128 = 128 2024-09-14T23:12:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-14T23:12:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:12:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-14T23:12:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_22': 2024-09-14T23:12:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:12:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:12:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:12:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 120fb18446c 2024-09-14T23:12:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.120fb18446c 2024-09-14T23:12:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:12:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:12:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:12:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:12:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:20 2024 2024-09-14T23:12:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:20 2024 2024-09-14T23:12:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:20 2024 2024-09-14T23:12:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:12:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:12:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:12:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: a9dd159e-4591-405f-b0f6-42ca7c8cdf26 2024-09-14T23:12:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:12:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-14T23:12:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:12:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:12:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_22.export 2024-09-14T23:12:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.export 2024-09-14T23:12:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_22.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.export 2024-09-14T23:12:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_22.export 2024-09-14T23:12:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.export 2024-09-14T23:12:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_22.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.export 2024-09-14T23:12:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_22.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_22.export 2024-09-14T23:12:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:12:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_23 2024-09-14T23:12:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-14T23:12:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:12:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-14T23:12:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-14T23:12:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:12:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-14T23:12:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:12:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:12:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:12:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:12:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:12:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:12:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-14T23:12:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-14T23:12:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:12:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:12:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:12:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-09-14T23:12:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-14T23:12:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:12:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:12:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:12:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:12:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-09-14T23:12:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:12:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:12:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:12:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:12:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:12:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-14T23:12:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_23.status 2024-09-14T23:12:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_23.status 2024-09-14T23:12:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-14T23:12:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_23.status 2024-09-14T23:12:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.status 2024-09-14T23:12:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-14T23:12:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-14T23:12:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 130 2024-09-14T23:12:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_23.status 2024-09-14T23:12:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_23.status 2024-09-14T23:12:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-14T23:12:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:12:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=130 2024-09-14T23:12:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:12:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.status 2024-09-14T23:12:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.status 2024-09-14T23:12:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-14T23:12:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 130 = '' 2024-09-14T23:12:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:12:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.status 2024-09-14T23:12:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.status 2024-09-14T23:12:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-14T23:12:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 130 = '' 2024-09-14T23:12:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:12:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.status 2024-09-14T23:12:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.status 2024-09-14T23:12:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-14T23:12:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=130 2024-09-14T23:12:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 130 = 130 2024-09-14T23:12:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-14T23:12:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:12:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-14T23:12:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_23': 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127e58a80f4d 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127e58a80f4d 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:29 2024 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:29 2024 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:29 2024 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:12:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:12:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: f42fac99-ab54-44f2-ac18-c845cd99d02b 2024-09-14T23:12:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:12:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-14T23:12:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:12:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:12:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_23.export 2024-09-14T23:12:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.export 2024-09-14T23:12:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_23.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.export 2024-09-14T23:12:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_23.export 2024-09-14T23:12:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.export 2024-09-14T23:12:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_23.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.export 2024-09-14T23:12:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_23.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_23.export 2024-09-14T23:12:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_24 2024-09-14T23:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-14T23:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:12:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-14T23:12:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:12:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:12:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:12:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-14T23:12:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:12:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:12:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:12:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:12:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:12:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:12:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:12:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:12:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:12:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:12:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:12:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-14T23:12:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-14T23:12:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:12:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:12:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:12:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-14T23:12:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-14T23:12:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:12:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:12:43.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:43.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:43.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:43.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-09-14T23:12:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:12:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:12:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:12:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:12:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:12:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:12:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:12:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-14T23:12:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_24.status 2024-09-14T23:12:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_24.status 2024-09-14T23:12:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-14T23:12:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_24.status 2024-09-14T23:12:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.status 2024-09-14T23:12:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-14T23:12:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-14T23:12:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 132 2024-09-14T23:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_24.status 2024-09-14T23:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_24.status 2024-09-14T23:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-14T23:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:12:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=132 2024-09-14T23:12:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:12:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.status 2024-09-14T23:12:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.status 2024-09-14T23:12:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-14T23:12:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 132 = '' 2024-09-14T23:12:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:12:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.status 2024-09-14T23:12:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.status 2024-09-14T23:12:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-14T23:12:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 132 = '' 2024-09-14T23:12:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:12:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.status 2024-09-14T23:12:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.status 2024-09-14T23:12:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-14T23:12:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=132 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 132 = 132 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:12:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-14T23:12:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_24': 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 128c358e88c1 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.128c358e88c1 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:36 2024 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:36 2024 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:36 2024 2024-09-14T23:12:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:12:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:12:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:12:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: d02fb2d0-2bf7-49a8-86d4-ecd3c3240c86 2024-09-14T23:12:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:12:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-14T23:12:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:12:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:12:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_24.export 2024-09-14T23:12:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.export 2024-09-14T23:12:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_24.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.export 2024-09-14T23:12:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_24.export 2024-09-14T23:12:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.export 2024-09-14T23:12:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_24.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.export 2024-09-14T23:12:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_24.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_24.export 2024-09-14T23:12:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:12:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_25 2024-09-14T23:12:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-14T23:12:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:12:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-14T23:12:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:12:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:12:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:12:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:12:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-14T23:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:12:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:12:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-14T23:12:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:12:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:12:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:12:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:12:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:12:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-14T23:12:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_25.status 2024-09-14T23:12:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_25.status 2024-09-14T23:12:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-14T23:12:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_25.status 2024-09-14T23:12:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.status 2024-09-14T23:12:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-14T23:12:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-14T23:12:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 134 2024-09-14T23:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_25.status 2024-09-14T23:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_25.status 2024-09-14T23:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-14T23:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:12:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=134 2024-09-14T23:12:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:12:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.status 2024-09-14T23:12:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.status 2024-09-14T23:12:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-14T23:12:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=59 2024-09-14T23:12:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 134 = 59 2024-09-14T23:12:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:12:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.status 2024-09-14T23:12:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.status 2024-09-14T23:12:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-14T23:12:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=59 2024-09-14T23:12:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 134 = 59 2024-09-14T23:12:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:12:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.status 2024-09-14T23:12:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.status 2024-09-14T23:12:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-14T23:12:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 134 = '' 2024-09-14T23:12:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:12:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.status 2024-09-14T23:12:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.status 2024-09-14T23:12:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-14T23:12:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=134 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 134 = 134 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:12:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-14T23:12:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_25': 2024-09-14T23:12:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:12:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e5b13307d2 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e5b13307d2 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:42 2024 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:42 2024 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:42 2024 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:12:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 16ceb1c6-1a5c-4b7b-8b20-da6200f477b6 2024-09-14T23:12:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:12:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-14T23:12:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:12:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:12:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_25.export 2024-09-14T23:12:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.export 2024-09-14T23:12:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_25.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.export 2024-09-14T23:12:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_25.export 2024-09-14T23:12:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.export 2024-09-14T23:12:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:12:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_25.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.export 2024-09-14T23:12:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_25.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_25.export 2024-09-14T23:12:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:12:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_26 2024-09-14T23:12:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-14T23:12:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:12:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:12:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-14T23:12:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-14T23:12:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:12:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:12:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:12:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-14T23:12:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:12:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:12:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:12:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-14T23:12:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:12:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:12:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:12:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:12:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:12:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:12:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:12:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:12:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:12:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-14T23:12:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:12:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:12:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:12:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:12:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-14T23:12:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-14T23:12:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-14T23:12:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:12:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:12:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:12:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:12:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:12:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-14T23:12:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-14T23:12:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:12:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:12:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-14T23:12:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-14T23:12:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:12:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:12:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:12:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-09-14T23:12:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:12:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:12:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:12:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:12:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:12:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:12:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:12:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:12:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:12:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:12:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:12:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:12:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-09-14T23:12:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:12:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:12:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:12:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:12:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:12:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-14T23:12:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:12:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-14T23:12:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_26.status 2024-09-14T23:12:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_26.status 2024-09-14T23:12:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-14T23:12:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_26.status 2024-09-14T23:12:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:12:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.status 2024-09-14T23:12:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-14T23:12:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-14T23:12:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 136 2024-09-14T23:12:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_26.status 2024-09-14T23:12:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:12:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_26.status 2024-09-14T23:12:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-14T23:12:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:12:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=136 2024-09-14T23:12:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:12:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.status 2024-09-14T23:12:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.status 2024-09-14T23:12:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-14T23:12:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=60 2024-09-14T23:12:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 136 = 60 2024-09-14T23:12:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:12:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.status 2024-09-14T23:12:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.status 2024-09-14T23:12:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-14T23:12:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=60 2024-09-14T23:12:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 136 = 60 2024-09-14T23:12:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:12:58.137 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T23:12:58.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T23:12:58.137 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-14T23:12:58.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:12:58.137+0000 7f701e168640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 55106) UID: 1000 2024-09-14T23:12:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.status 2024-09-14T23:12:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:12:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:12:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:12:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.status 2024-09-14T23:12:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-14T23:12:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:12:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:12:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:12:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:12:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:12:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:12:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:12:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 136 = '' 2024-09-14T23:12:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:12:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:13:00.139 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T23:13:00.139 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:13:00.139 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T23:13:00.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T23:13:00.139 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T23:13:00.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T23:13:00.142 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-09-14T23:13:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.status 2024-09-14T23:13:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:13:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.status 2024-09-14T23:13:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-14T23:13:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=136 2024-09-14T23:13:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 136 = 136 2024-09-14T23:13:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-14T23:13:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:13:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:13:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_26': 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 120f2800ca1c 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.120f2800ca1c 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:04:57 2024 2024-09-14T23:13:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:04:57 2024 2024-09-14T23:13:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:04:57 2024 2024-09-14T23:13:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:13:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:13:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:13:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 68d375d5-545a-4f5c-87a8-2ef5ae048705 2024-09-14T23:13:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:13:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-14T23:13:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:13:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:13:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_26.export 2024-09-14T23:13:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.export 2024-09-14T23:13:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_26.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.export 2024-09-14T23:13:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_26.export 2024-09-14T23:13:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.export 2024-09-14T23:13:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_26.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.export 2024-09-14T23:13:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_26.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_26.export 2024-09-14T23:13:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:13:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_27 2024-09-14T23:13:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-14T23:13:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:13:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-14T23:13:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-14T23:13:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:13:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-14T23:13:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:13:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:13:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:13:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-14T23:13:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:13:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:13:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:13:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:13:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:13:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:13:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-14T23:13:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-14T23:13:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-14T23:13:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:13:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:13:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:13:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:13:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-14T23:13:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-14T23:13:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:13:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:13:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:13:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:13:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:13:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:13:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:13:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:13:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-14T23:13:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:13:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:13:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:13:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-09-14T23:13:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 1 2024-09-14T23:13:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-14T23:13:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-14T23:13:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:13:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:13:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:13:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:13:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:13:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-14T23:13:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-14T23:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:13:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:13:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:13:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:13:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-14T23:13:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:13:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:13:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:13:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-09-14T23:13:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 2 2024-09-14T23:13:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-14T23:13:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-14T23:13:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:13:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:13:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:13:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:13:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:13:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:13:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-14T23:13:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:13:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:13:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:13:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-14T23:13:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-09-14T23:13:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:13:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:13:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:13:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:13:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:13:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:13:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-09-14T23:13:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 4 2024-09-14T23:13:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-14T23:13:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:13:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:13:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:13:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:13:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-14T23:13:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-14T23:13:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:13:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:13:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:13:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:13:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:13:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-14T23:13:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-09-14T23:13:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:13:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:13:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:13:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:13:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:13:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-09-14T23:13:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T23:13:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-14T23:13:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:13:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:13:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:13:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-14T23:13:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-14T23:13:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:13:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:13:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:13:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:13:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:13:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:13:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:13:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-14T23:13:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:13:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:13:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:13:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-09-14T23:13:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-14T23:13:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T23:13:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-14T23:13:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:13:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Starting"' 2024-09-14T23:13:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Starting"' 2024-09-14T23:13:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:13:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = stopped 2024-09-14T23:13:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:13:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-14T23:13:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:13:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:13:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:13:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-14T23:13:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:13:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:13:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:13:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:13:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:13:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-14T23:13:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:13:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-14T23:13:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_27.status 2024-09-14T23:13:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:13:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_27.status 2024-09-14T23:13:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-14T23:13:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_27.status 2024-09-14T23:13:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:13:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.status 2024-09-14T23:13:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-14T23:13:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-14T23:13:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 138 2024-09-14T23:13:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_27.status 2024-09-14T23:13:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_27.status 2024-09-14T23:13:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-14T23:13:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:13:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=138 2024-09-14T23:13:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:13:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:13:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.status 2024-09-14T23:13:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.status 2024-09-14T23:13:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-14T23:13:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=61 2024-09-14T23:13:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 138 = 61 2024-09-14T23:13:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:13:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.status 2024-09-14T23:13:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.status 2024-09-14T23:13:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-14T23:13:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=61 2024-09-14T23:13:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 138 = 61 2024-09-14T23:13:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:13:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.status 2024-09-14T23:13:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.status 2024-09-14T23:13:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-14T23:13:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:13:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:13:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:13:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:13:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:13:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 138 = '' 2024-09-14T23:13:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:13:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.status 2024-09-14T23:13:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.status 2024-09-14T23:13:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-14T23:13:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=138 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 138 = 138 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:13:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-14T23:13:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_27': 2024-09-14T23:13:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:13:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:13:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127ec634c0a0 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127ec634c0a0 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:03 2024 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:03 2024 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:03 2024 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: a848ce88-5df8-4b90-b6a1-72b3261b72bc 2024-09-14T23:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:13:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-14T23:13:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:13:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:13:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_27.export 2024-09-14T23:13:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.export 2024-09-14T23:13:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_27.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.export 2024-09-14T23:13:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_27.export 2024-09-14T23:13:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.export 2024-09-14T23:13:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_27.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.export 2024-09-14T23:13:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_27.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_27.export 2024-09-14T23:13:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:13:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_28 2024-09-14T23:13:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-14T23:13:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-14T23:13:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:13:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:13:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:13:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-14T23:13:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:13:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:13:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:13:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:13:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:13:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:13:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-14T23:13:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-14T23:13:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-14T23:13:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-14T23:13:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:13:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:13:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:13:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:13:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:13:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:13:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-14T23:13:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:13:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-14T23:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_28.status 2024-09-14T23:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_28.status 2024-09-14T23:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-14T23:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_28.status 2024-09-14T23:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:13:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.status 2024-09-14T23:13:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-14T23:13:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-14T23:13:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 140 2024-09-14T23:13:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_28.status 2024-09-14T23:13:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_28.status 2024-09-14T23:13:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-14T23:13:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:13:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=140 2024-09-14T23:13:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:13:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:13:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.status 2024-09-14T23:13:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.status 2024-09-14T23:13:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-14T23:13:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=62 2024-09-14T23:13:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 140 = 62 2024-09-14T23:13:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:13:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.status 2024-09-14T23:13:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.status 2024-09-14T23:13:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-14T23:13:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=62 2024-09-14T23:13:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 140 = 62 2024-09-14T23:13:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:13:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.status 2024-09-14T23:13:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.status 2024-09-14T23:13:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-14T23:13:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:13:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:13:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:13:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:13:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:13:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 140 = '' 2024-09-14T23:13:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:13:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.status 2024-09-14T23:13:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.status 2024-09-14T23:13:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-14T23:13:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=140 2024-09-14T23:13:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 140 = 140 2024-09-14T23:13:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-14T23:13:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:13:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-14T23:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_28': 2024-09-14T23:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 128cfbd7d148 2024-09-14T23:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.128cfbd7d148 2024-09-14T23:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:18 2024 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:18 2024 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:18 2024 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 4c9c9805-ba1a-48da-a08b-8de70de557b3 2024-09-14T23:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:13:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-14T23:13:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:13:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:13:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_28.export 2024-09-14T23:13:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.export 2024-09-14T23:13:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_28.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.export 2024-09-14T23:13:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_28.export 2024-09-14T23:13:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.export 2024-09-14T23:13:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_28.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.export 2024-09-14T23:13:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_28.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_28.export 2024-09-14T23:13:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:13:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_29 2024-09-14T23:13:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-14T23:13:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:13:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-14T23:13:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-14T23:13:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:13:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-14T23:13:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:13:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:13:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:13:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-14T23:13:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:13:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:13:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:13:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:13:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:13:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:13:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-14T23:13:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-14T23:13:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-14T23:13:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:13:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:13:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:13:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-09-14T23:13:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-14T23:13:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-09-14T23:13:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:13:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:13:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:13:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:13:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-09-14T23:13:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:13:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:13:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:13:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:13:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:13:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-14T23:13:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:13:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-14T23:13:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_29.status 2024-09-14T23:13:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:13:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_29.status 2024-09-14T23:13:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-14T23:13:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_29.status 2024-09-14T23:13:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:13:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.status 2024-09-14T23:13:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-14T23:13:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-14T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 142 2024-09-14T23:13:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_29.status 2024-09-14T23:13:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_29.status 2024-09-14T23:13:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-14T23:13:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:13:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=142 2024-09-14T23:13:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:13:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:13:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.status 2024-09-14T23:13:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.status 2024-09-14T23:13:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-14T23:13:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=63 2024-09-14T23:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 142 = 63 2024-09-14T23:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:13:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.status 2024-09-14T23:13:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.status 2024-09-14T23:13:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-14T23:13:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=63 2024-09-14T23:13:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 142 = 63 2024-09-14T23:13:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:13:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.status 2024-09-14T23:13:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.status 2024-09-14T23:13:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-14T23:13:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:13:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:13:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:13:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:13:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:13:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 142 = '' 2024-09-14T23:13:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:13:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.status 2024-09-14T23:13:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.status 2024-09-14T23:13:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-14T23:13:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=142 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 142 = 142 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:13:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-14T23:13:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_29': 2024-09-14T23:13:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:13:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:13:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:13:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e57e88d34 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e57e88d34 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:33 2024 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:33 2024 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:33 2024 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: ad5acb85-55a5-4fcf-abcf-6bf6f2766d18 2024-09-14T23:13:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:13:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-14T23:13:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:13:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:13:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_29.export 2024-09-14T23:13:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.export 2024-09-14T23:13:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_29.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.export 2024-09-14T23:13:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_29.export 2024-09-14T23:13:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.export 2024-09-14T23:13:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_29.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.export 2024-09-14T23:13:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_29.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_29.export 2024-09-14T23:13:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:13:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_30 2024-09-14T23:13:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-14T23:13:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:13:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-14T23:13:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-14T23:13:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:13:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:13:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:13:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-14T23:13:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:13:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:13:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:13:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:13:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:13:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:13:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-14T23:13:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:13:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:13:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:13:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:13:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-14T23:13:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:13:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-14T23:13:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_30.status 2024-09-14T23:13:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:13:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_30.status 2024-09-14T23:13:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-14T23:13:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_30.status 2024-09-14T23:13:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:13:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.status 2024-09-14T23:13:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-14T23:13:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-14T23:13:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 144 2024-09-14T23:13:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_30.status 2024-09-14T23:13:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_30.status 2024-09-14T23:13:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-14T23:13:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:13:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=144 2024-09-14T23:13:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:13:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:13:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.status 2024-09-14T23:13:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.status 2024-09-14T23:13:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-14T23:13:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=64 2024-09-14T23:13:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 144 = 64 2024-09-14T23:13:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:13:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.status 2024-09-14T23:13:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.status 2024-09-14T23:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-14T23:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=64 2024-09-14T23:13:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 144 = 64 2024-09-14T23:13:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:13:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.status 2024-09-14T23:13:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.status 2024-09-14T23:13:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-14T23:13:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:13:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:13:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:13:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:13:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:13:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 144 = '' 2024-09-14T23:13:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:13:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:13:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.status 2024-09-14T23:13:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.status 2024-09-14T23:13:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-14T23:13:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:13:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:13:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=144 2024-09-14T23:13:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 144 = 144 2024-09-14T23:13:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-14T23:13:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:13:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-14T23:13:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_30': 2024-09-14T23:13:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:13:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:13:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:13:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 120f61b6c3be 2024-09-14T23:13:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.120f61b6c3be 2024-09-14T23:13:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:13:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:13:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:13:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:13:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:40 2024 2024-09-14T23:13:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:40 2024 2024-09-14T23:13:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:40 2024 2024-09-14T23:13:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:13:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:13:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:13:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 2dd3a48d-e1ef-4b2b-b6fb-173793741ecc 2024-09-14T23:13:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:13:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-14T23:13:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:13:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:13:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_30.export 2024-09-14T23:13:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.export 2024-09-14T23:13:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_30.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.export 2024-09-14T23:13:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_30.export 2024-09-14T23:13:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.export 2024-09-14T23:13:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:13:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_30.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.export 2024-09-14T23:13:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_30.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_30.export 2024-09-14T23:13:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:13:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:13:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_31 2024-09-14T23:13:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-14T23:13:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:13:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:13:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:13:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:13:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-14T23:13:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:13:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-14T23:13:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:13:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:13:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:13:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:13:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:13:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:13:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-14T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:13:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:13:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-14T23:13:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:13:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:13:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:13:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:13:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:13:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:13:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:13:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:13:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:13:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:13:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-14T23:13:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-14T23:13:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-09-14T23:13:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:13:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:13:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:13:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:13:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:13:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-09-14T23:13:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-14T23:13:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-09-14T23:13:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:13:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:13:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:13:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:13:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:13:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-09-14T23:13:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-14T23:13:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:13:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_31 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:13:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:13:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:13:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:13:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:13:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:13:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:13:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:13:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:13:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:13:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_31 2024-09-14T23:14:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:14:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:14:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:14:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:14:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:14:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:14:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:14:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-14T23:14:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:14:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_31.status 2024-09-14T23:14:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:14:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_31.status 2024-09-14T23:14:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-14T23:14:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_31.status 2024-09-14T23:14:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:14:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.status 2024-09-14T23:14:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-14T23:14:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-14T23:14:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 146 2024-09-14T23:14:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_31.status 2024-09-14T23:14:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_31.status 2024-09-14T23:14:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-14T23:14:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:14:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=146 2024-09-14T23:14:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:14:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.status 2024-09-14T23:14:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.status 2024-09-14T23:14:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-14T23:14:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=65 2024-09-14T23:14:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 146 = 65 2024-09-14T23:14:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:14:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.status 2024-09-14T23:14:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.status 2024-09-14T23:14:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-14T23:14:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:14:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:14:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:14:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:14:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 146 = '' 2024-09-14T23:14:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:14:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.status 2024-09-14T23:14:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.status 2024-09-14T23:14:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-14T23:14:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:14:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:14:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:14:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:14:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 146 = '' 2024-09-14T23:14:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:14:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.status 2024-09-14T23:14:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.status 2024-09-14T23:14:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-14T23:14:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=146 2024-09-14T23:14:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 146 = 146 2024-09-14T23:14:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-14T23:14:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:14:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:14:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:14:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-14T23:14:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_31': 2024-09-14T23:14:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:14:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:14:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:14:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127e9f1bd109 2024-09-14T23:14:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127e9f1bd109 2024-09-14T23:14:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:14:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:14:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:14:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:14:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:05:51 2024 2024-09-14T23:14:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:05:51 2024 2024-09-14T23:14:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:05:51 2024 2024-09-14T23:14:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:14:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:14:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:14:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: d5f8f61a-4276-4546-8e0b-db256cf1d59a 2024-09-14T23:14:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:14:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-14T23:14:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:14:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:14:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_31.export 2024-09-14T23:14:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.export 2024-09-14T23:14:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_31.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.export 2024-09-14T23:14:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_31.export 2024-09-14T23:14:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:14:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.export 2024-09-14T23:14:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:14:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_31.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.export 2024-09-14T23:14:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_31.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_31.export 2024-09-14T23:14:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:14:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_32 2024-09-14T23:14:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-14T23:14:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:14:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:14:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-14T23:14:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:14:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:14:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:14:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:14:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:14:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:14:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:14:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:14:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:14:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:14:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:14:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:14:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-14T23:14:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-14T23:14:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-14T23:14:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:14:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:14:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:14:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-14T23:14:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-14T23:14:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:14:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:14:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:14:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:14:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2024-09-14T23:14:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:14:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:14:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:14:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:14:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:14:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:14:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:14:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-14T23:14:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-14T23:14:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_32.status 2024-09-14T23:14:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:14:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_32.status 2024-09-14T23:14:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-14T23:14:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_32.status 2024-09-14T23:14:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:14:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.status 2024-09-14T23:14:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-14T23:14:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-14T23:14:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 148 2024-09-14T23:14:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_32.status 2024-09-14T23:14:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_32.status 2024-09-14T23:14:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-14T23:14:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:14:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=148 2024-09-14T23:14:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:14:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.status 2024-09-14T23:14:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.status 2024-09-14T23:14:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-14T23:14:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=66 2024-09-14T23:14:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 148 = 66 2024-09-14T23:14:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:14:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.status 2024-09-14T23:14:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.status 2024-09-14T23:14:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-14T23:14:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:14:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:14:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:14:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:14:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 148 = '' 2024-09-14T23:14:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:14:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.status 2024-09-14T23:14:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.status 2024-09-14T23:14:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-14T23:14:09.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:14:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:14:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:14:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:14:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 148 = '' 2024-09-14T23:14:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:14:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.status 2024-09-14T23:14:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.status 2024-09-14T23:14:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-14T23:14:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=148 2024-09-14T23:14:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 148 = 148 2024-09-14T23:14:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-14T23:14:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:14:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:14:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:14:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-14T23:14:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_32': 2024-09-14T23:14:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:14:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 128c6c99eb5 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.128c6c99eb5 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:01 2024 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:01 2024 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:01 2024 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:14:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 79596924-7c99-4578-ab4b-9b96094cd7b2 2024-09-14T23:14:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:14:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-14T23:14:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:14:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:14:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_32.export 2024-09-14T23:14:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.export 2024-09-14T23:14:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_32.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.export 2024-09-14T23:14:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_32.export 2024-09-14T23:14:12.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:14:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.export 2024-09-14T23:14:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:14:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_32.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.export 2024-09-14T23:14:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_32.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_32.export 2024-09-14T23:14:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:14:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_33 2024-09-14T23:14:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-14T23:14:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:14:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-14T23:14:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-14T23:14:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:14:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:14:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:14:13.143 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T23:14:13.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T23:14:13.144 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T23:14:13.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:14:13.144+0000 7f0cbc6f2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 57109) UID: 1000 2024-09-14T23:14:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-14T23:14:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:14:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:14:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:14:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:14:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:14:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:14:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:14:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:14:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:14:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:14:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:14:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:14:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-14T23:14:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:14:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:14:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:14:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-14T23:14:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:14:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:14:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:14:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-14T23:14:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:14:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:14:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:14:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-09-14T23:14:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 1 2024-09-14T23:14:14.146 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T23:14:14.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T23:14:14.146 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-14T23:14:14.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:14:14.146+0000 7fc867432640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 42733) UID: 1000 2024-09-14T23:14:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:14:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:14:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-14T23:14:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:14:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:14:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:14:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:14:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-14T23:14:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-14T23:14:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-14T23:14:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:14:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:14:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:14:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:14:15.177 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T23:14:15.176+0000 7f5fe0b61640 -1 reset not still connected to 0x559280f40b60 2024-09-14T23:14:15.180 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T23:14:15.176+0000 7f5fe0b61640 -1 reset not still connected to 0x559283caf790 2024-09-14T23:14:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-14T23:14:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-14T23:14:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:14:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:14:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:14:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:14:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:14:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:14:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:14:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-14T23:14:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:14:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:14:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:14:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-09-14T23:14:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 2 2024-09-14T23:14:17.147 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T23:14:17.147 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:14:17.147 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T23:14:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T23:14:17.148 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T23:14:17.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T23:14:17.151 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-14T23:14:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:14:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:14:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-14T23:14:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:14:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:14:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:14:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:14:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:14:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-14T23:14:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-14T23:14:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:14:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:14:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:14:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-14T23:14:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:14:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:14:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:14:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:14:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-14T23:14:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-09-14T23:14:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:14:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:14:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-09-14T23:14:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 4 2024-09-14T23:14:18.144 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T23:14:18.145 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:14:18.145 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T23:14:18.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T23:14:18.145 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T23:14:18.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T23:14:18.148 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-14T23:14:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:14:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:14:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-14T23:14:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:14:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:14:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:14:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:14:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-14T23:14:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-14T23:14:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:14:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:14:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:14:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:14:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-14T23:14:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-14T23:14:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:14:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:14:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:14:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:14:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:14:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-14T23:14:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-09-14T23:14:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:14:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:14:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:14:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:14:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-09-14T23:14:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T23:14:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:14:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:14:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-14T23:14:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:14:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:14:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:14:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:14:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:14:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:14:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:14:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:14:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-14T23:14:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-14T23:14:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:14:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:14:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:14:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:14:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:14:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-14T23:14:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:14:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:14:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:14:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-14T23:14:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-09-14T23:14:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:14:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:14:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:14:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:14:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:14:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-09-14T23:14:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T23:14:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:14:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:14:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-14T23:14:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:14:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:14:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:14:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:14:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-14T23:14:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-14T23:14:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:14:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:14:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:14:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:14:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-14T23:14:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:14:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:14:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:14:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-14T23:14:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:14:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:14:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:14:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:14:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-09-14T23:14:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 8 2024-09-14T23:14:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:14:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:14:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-14T23:14:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:14:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:14:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:14:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:14:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:14:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:14:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:14:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-14T23:14:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-14T23:14:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:14:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:14:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:14:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:14:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-14T23:14:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:14:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:14:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:14:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-14T23:14:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-14T23:14:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:14:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:14:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:14:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-09-14T23:14:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:14:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:14:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:14:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:14:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:14:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-14T23:14:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_33.status 2024-09-14T23:14:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:14:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_33.status 2024-09-14T23:14:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-14T23:14:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_33.status 2024-09-14T23:14:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:14:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.status 2024-09-14T23:14:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-14T23:14:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-14T23:14:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 150 2024-09-14T23:14:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_33.status 2024-09-14T23:14:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_33.status 2024-09-14T23:14:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-14T23:14:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:14:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=150 2024-09-14T23:14:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:14:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.status 2024-09-14T23:14:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.status 2024-09-14T23:14:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-14T23:14:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=67 2024-09-14T23:14:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 150 = 67 2024-09-14T23:14:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:14:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.status 2024-09-14T23:14:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.status 2024-09-14T23:14:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-14T23:14:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=67 2024-09-14T23:14:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 150 = 67 2024-09-14T23:14:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:14:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.status 2024-09-14T23:14:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.status 2024-09-14T23:14:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-14T23:14:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:14:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:14:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:14:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:14:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 150 = '' 2024-09-14T23:14:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:14:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.status 2024-09-14T23:14:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.status 2024-09-14T23:14:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-14T23:14:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=150 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 150 = 150 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:14:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:14:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-14T23:14:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_33': 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e59f27bb25 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e59f27bb25 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:09 2024 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:09 2024 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:09 2024 2024-09-14T23:14:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:14:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:14:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:14:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 4c528d36-e484-45fe-9609-82252ec0ca3f 2024-09-14T23:14:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:14:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-14T23:14:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:14:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:14:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_33.export 2024-09-14T23:14:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.export 2024-09-14T23:14:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_33.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.export 2024-09-14T23:14:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_33.export 2024-09-14T23:14:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:14:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.export 2024-09-14T23:14:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:14:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_33.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.export 2024-09-14T23:14:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_33.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_33.export 2024-09-14T23:14:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:14:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_34 2024-09-14T23:14:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-14T23:14:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:14:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-14T23:14:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-14T23:14:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:14:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:14:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:14:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:14:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:14:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:14:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:14:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:14:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:14:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:14:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:14:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-14T23:14:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:14:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:14:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:14:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:14:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:14:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:14:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:14:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:14:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:14:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-14T23:14:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-14T23:14:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:14:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:14:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:14:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:14:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:14:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:14:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-14T23:14:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-14T23:14:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2024-09-14T23:14:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:14:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:14:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:14:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:14:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:14:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:14:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2024-09-14T23:14:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-14T23:14:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:14:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:14:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:14:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:14:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:14:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:14:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:14:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:14:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:14:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:14:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:14:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:14:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:14:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2024-09-14T23:14:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:14:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:14:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:14:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:14:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:14:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:14:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:14:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-14T23:14:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_34.status 2024-09-14T23:14:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:14:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_34.status 2024-09-14T23:14:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-14T23:14:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_34.status 2024-09-14T23:14:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:14:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.status 2024-09-14T23:14:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-14T23:14:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-14T23:14:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 152 2024-09-14T23:14:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_34.status 2024-09-14T23:14:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:14:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_34.status 2024-09-14T23:14:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-14T23:14:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:14:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=152 2024-09-14T23:14:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:14:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.status 2024-09-14T23:14:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.status 2024-09-14T23:14:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-14T23:14:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=68 2024-09-14T23:14:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 152 = 68 2024-09-14T23:14:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:14:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.status 2024-09-14T23:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.status 2024-09-14T23:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-14T23:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=68 2024-09-14T23:14:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 152 = 68 2024-09-14T23:14:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:14:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.status 2024-09-14T23:14:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.status 2024-09-14T23:14:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-14T23:14:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:14:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:14:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:14:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:14:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:14:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 152 = '' 2024-09-14T23:14:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:14:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:14:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.status 2024-09-14T23:14:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.status 2024-09-14T23:14:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-14T23:14:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:14:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:14:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=152 2024-09-14T23:14:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 152 = 152 2024-09-14T23:14:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-14T23:14:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:14:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:14:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:14:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:14:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-14T23:14:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_34': 2024-09-14T23:14:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:14:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:14:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:14:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127eddb6534 2024-09-14T23:14:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127eddb6534 2024-09-14T23:14:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:14:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:14:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:14:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:14:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:50 2024 2024-09-14T23:14:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:50 2024 2024-09-14T23:14:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:50 2024 2024-09-14T23:14:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:14:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:14:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:14:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: b3b49072-90fc-4a20-87f1-ba0c9f24cea9 2024-09-14T23:14:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:14:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:14:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-14T23:14:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:14:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:14:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:14:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:14:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_34.export 2024-09-14T23:14:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.export 2024-09-14T23:14:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_34.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.export 2024-09-14T23:14:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_34.export 2024-09-14T23:15:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.export 2024-09-14T23:15:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_34.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.export 2024-09-14T23:15:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_34.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_34.export 2024-09-14T23:15:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_35 2024-09-14T23:15:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-14T23:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-14T23:15:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-14T23:15:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-14T23:15:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:15:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:15:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:15:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-14T23:15:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:15:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:15:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:15:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:15:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:15:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:15:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:15:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:15:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:15:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:15:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:15:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-14T23:15:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-14T23:15:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:15:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:15:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:15:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-09-14T23:15:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-14T23:15:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2024-09-14T23:15:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:15:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:15:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:15:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:15:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2024-09-14T23:15:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-14T23:15:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_35 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:15:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:15:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:15:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_35 2024-09-14T23:15:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:15:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:15:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:15:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:15:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:15:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:15:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:15:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-14T23:15:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_35.status 2024-09-14T23:15:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_35.status 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_35.status 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.status 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-14T23:15:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 154 2024-09-14T23:15:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_35.status 2024-09-14T23:15:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_35.status 2024-09-14T23:15:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-14T23:15:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:15:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=154 2024-09-14T23:15:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:15:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.status 2024-09-14T23:15:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.status 2024-09-14T23:15:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-14T23:15:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=69 2024-09-14T23:15:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 154 = 69 2024-09-14T23:15:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:15:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.status 2024-09-14T23:15:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.status 2024-09-14T23:15:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-14T23:15:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=69 2024-09-14T23:15:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 154 = 69 2024-09-14T23:15:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:15:04.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.status 2024-09-14T23:15:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.status 2024-09-14T23:15:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-14T23:15:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 154 = '' 2024-09-14T23:15:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:15:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.status 2024-09-14T23:15:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.status 2024-09-14T23:15:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-14T23:15:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=154 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 154 = 154 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:15:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-14T23:15:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_35': 2024-09-14T23:15:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:15:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:15:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12a01bb661b4 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12a01bb661b4 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:30 2024 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:30 2024 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:30 2024 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 62edd50a-04e9-49e3-95e0-0652ce1bea79 2024-09-14T23:15:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:15:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-14T23:15:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:15:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:15:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_35.export 2024-09-14T23:15:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.export 2024-09-14T23:15:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_35.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.export 2024-09-14T23:15:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_35.export 2024-09-14T23:15:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.export 2024-09-14T23:15:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_35.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.export 2024-09-14T23:15:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_35.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_35.export 2024-09-14T23:15:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_36 2024-09-14T23:15:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-14T23:15:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-14T23:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-14T23:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-14T23:15:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:15:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:15:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:15:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:15:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:15:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-14T23:15:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:15:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:15:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:15:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:15:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:15:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:15:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-14T23:15:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-14T23:15:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-14T23:15:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:15:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:15:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:15:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:15:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:15:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-14T23:15:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-14T23:15:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:15:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:15:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:15:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-14T23:15:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-14T23:15:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:15:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:15:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:15:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-14T23:15:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:15:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:15:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:15:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:15:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-14T23:15:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_36.status 2024-09-14T23:15:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_36.status 2024-09-14T23:15:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-14T23:15:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_36.status 2024-09-14T23:15:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.status 2024-09-14T23:15:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-14T23:15:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-14T23:15:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 156 2024-09-14T23:15:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_36.status 2024-09-14T23:15:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_36.status 2024-09-14T23:15:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-14T23:15:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:15:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=156 2024-09-14T23:15:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:15:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.status 2024-09-14T23:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.status 2024-09-14T23:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-14T23:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=70 2024-09-14T23:15:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 156 = 70 2024-09-14T23:15:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:15:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.status 2024-09-14T23:15:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.status 2024-09-14T23:15:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-14T23:15:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 156 = '' 2024-09-14T23:15:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:15:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.status 2024-09-14T23:15:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.status 2024-09-14T23:15:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-14T23:15:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=156 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 156 = 156 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:15:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-14T23:15:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_36': 2024-09-14T23:15:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:15:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12a0afc9d3a 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12a0afc9d3a 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:37 2024 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:37 2024 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:37 2024 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:15:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:15:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 9def2b16-9e96-4962-a4ee-3a7a75bafdd6 2024-09-14T23:15:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:15:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-14T23:15:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:15:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:15:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_36.export 2024-09-14T23:15:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.export 2024-09-14T23:15:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_36.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.export 2024-09-14T23:15:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_36.export 2024-09-14T23:15:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.export 2024-09-14T23:15:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_36.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.export 2024-09-14T23:15:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_36.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_36.export 2024-09-14T23:15:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_37 2024-09-14T23:15:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-14T23:15:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-14T23:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-14T23:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-14T23:15:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:15:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:15:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:15:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-14T23:15:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:15:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:15:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:15:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:15:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:15:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:15:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:15:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:15:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:15:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-14T23:15:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-14T23:15:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-09-14T23:15:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:15:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:15:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:15:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-09-14T23:15:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-14T23:15:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-09-14T23:15:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:15:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:15:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:15:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:15:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:15:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-09-14T23:15:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-14T23:15:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_37 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:15:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:15:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:15:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_37 2024-09-14T23:15:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:15:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:15:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:15:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:15:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:15:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:15:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:15:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-14T23:15:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:15:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:15:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-14T23:15:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-14T23:15:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_37.status 2024-09-14T23:15:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_37.status 2024-09-14T23:15:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-14T23:15:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_37.status 2024-09-14T23:15:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.status 2024-09-14T23:15:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-14T23:15:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-14T23:15:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 158 2024-09-14T23:15:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_37.status 2024-09-14T23:15:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_37.status 2024-09-14T23:15:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-14T23:15:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:15:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=158 2024-09-14T23:15:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:15:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.status 2024-09-14T23:15:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.status 2024-09-14T23:15:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-14T23:15:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=71 2024-09-14T23:15:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 158 = 71 2024-09-14T23:15:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:15:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.status 2024-09-14T23:15:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.status 2024-09-14T23:15:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-14T23:15:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=71 2024-09-14T23:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 158 = 71 2024-09-14T23:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:15:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.status 2024-09-14T23:15:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.status 2024-09-14T23:15:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-14T23:15:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 158 = '' 2024-09-14T23:15:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:15:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.status 2024-09-14T23:15:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.status 2024-09-14T23:15:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-14T23:15:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=158 2024-09-14T23:15:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 158 = 158 2024-09-14T23:15:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-14T23:15:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:15:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-14T23:15:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_37': 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12a02d5e2b24 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12a02d5e2b24 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:06:48 2024 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:06:48 2024 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:06:48 2024 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:15:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:15:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:15:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: efd468e5-5b78-4874-94e4-101d543ad42f 2024-09-14T23:15:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:15:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-14T23:15:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:15:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:15:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_37.export 2024-09-14T23:15:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.export 2024-09-14T23:15:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_37.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.export 2024-09-14T23:15:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_37.export 2024-09-14T23:15:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.export 2024-09-14T23:15:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_37.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.export 2024-09-14T23:15:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_37.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_37.export 2024-09-14T23:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_38 2024-09-14T23:15:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-14T23:15:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-14T23:15:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-14T23:15:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-14T23:15:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:15:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:15:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:15:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-14T23:15:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:15:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:15:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:15:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:15:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:15:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:15:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:15:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:15:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-14T23:15:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-14T23:15:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:15:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:15:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:15:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-14T23:15:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-14T23:15:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2024-09-14T23:15:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:15:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:15:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:15:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:15:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2024-09-14T23:15:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-14T23:15:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:15:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:15:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:15:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:15:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2024-09-14T23:15:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:15:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:15:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:15:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:15:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:15:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:15:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:15:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-14T23:15:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_38.status 2024-09-14T23:15:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_38.status 2024-09-14T23:15:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-14T23:15:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_38.status 2024-09-14T23:15:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.status 2024-09-14T23:15:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-14T23:15:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-14T23:15:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 160 2024-09-14T23:15:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_38.status 2024-09-14T23:15:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_38.status 2024-09-14T23:15:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-14T23:15:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:15:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=160 2024-09-14T23:15:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:15:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.status 2024-09-14T23:15:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.status 2024-09-14T23:15:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-14T23:15:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=72 2024-09-14T23:15:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 160 = 72 2024-09-14T23:15:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:15:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.status 2024-09-14T23:15:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.status 2024-09-14T23:15:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-14T23:15:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 160 = '' 2024-09-14T23:15:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:15:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.status 2024-09-14T23:15:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.status 2024-09-14T23:15:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-14T23:15:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 160 = '' 2024-09-14T23:15:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:15:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.status 2024-09-14T23:15:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.status 2024-09-14T23:15:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-14T23:15:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=160 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 160 = 160 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:15:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:26.152 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4006894776688531) > weight (0.25) 2024-09-14T23:15:26.152 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-14T23:15:26.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-14T23:15:26.153 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-14T23:15:26.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:15:26.152+0000 7f3cba3a9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 55786) UID: 1000 2024-09-14T23:15:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-14T23:15:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_38': 2024-09-14T23:15:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:15:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127e7f64ea9a 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127e7f64ea9a 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:02 2024 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:02 2024 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:02 2024 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:15:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 3f0fd191-d8d6-479a-bc37-05f42708eaf7 2024-09-14T23:15:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:15:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-14T23:15:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:15:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:15:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_38.export 2024-09-14T23:15:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.export 2024-09-14T23:15:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_38.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.export 2024-09-14T23:15:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_38.export 2024-09-14T23:15:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.export 2024-09-14T23:15:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_38.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.export 2024-09-14T23:15:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_38.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_38.export 2024-09-14T23:15:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_39 2024-09-14T23:15:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-14T23:15:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-14T23:15:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-14T23:15:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-14T23:15:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:15:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:15:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:15:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:15:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:15:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:15:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:15:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:15:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-14T23:15:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-14T23:15:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:15:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:15:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:15:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-14T23:15:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:15:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:15:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:15:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:15:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:15:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:15:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:15:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-14T23:15:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_39.status 2024-09-14T23:15:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_39.status 2024-09-14T23:15:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-14T23:15:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_39.status 2024-09-14T23:15:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.status 2024-09-14T23:15:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-14T23:15:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-14T23:15:29.153 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-14T23:15:29.153 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:15:29.154 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-14T23:15:29.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T23:15:29.154 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T23:15:29.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T23:15:29.160 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-09-14T23:15:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 162 2024-09-14T23:15:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_39.status 2024-09-14T23:15:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_39.status 2024-09-14T23:15:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-14T23:15:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:15:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=162 2024-09-14T23:15:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.status 2024-09-14T23:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.status 2024-09-14T23:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T23:15:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=73 2024-09-14T23:15:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 162 = 73 2024-09-14T23:15:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:15:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.status 2024-09-14T23:15:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.status 2024-09-14T23:15:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T23:15:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=73 2024-09-14T23:15:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 162 = 73 2024-09-14T23:15:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:15:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.status 2024-09-14T23:15:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.status 2024-09-14T23:15:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T23:15:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 162 = '' 2024-09-14T23:15:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:15:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.status 2024-09-14T23:15:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.status 2024-09-14T23:15:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T23:15:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=162 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 162 = 162 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:15:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-14T23:15:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_39': 2024-09-14T23:15:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:15:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:15:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:15:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 128c1dcbcef1 2024-09-14T23:15:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.128c1dcbcef1 2024-09-14T23:15:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:15:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:15:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:15:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:15:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:08 2024 2024-09-14T23:15:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:08 2024 2024-09-14T23:15:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:08 2024 2024-09-14T23:15:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:15:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:15:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:15:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 2c75e949-74f5-488e-aeb1-ce0ee7bef852 2024-09-14T23:15:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:15:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-14T23:15:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:15:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:15:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_39.export 2024-09-14T23:15:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.export 2024-09-14T23:15:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_39.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.export 2024-09-14T23:15:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_39.export 2024-09-14T23:15:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.export 2024-09-14T23:15:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_39.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.export 2024-09-14T23:15:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_39.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_39.export 2024-09-14T23:15:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_40 2024-09-14T23:15:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-14T23:15:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-14T23:15:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:15:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:15:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:15:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-14T23:15:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:15:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:15:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:15:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:15:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:15:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:15:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:15:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:15:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-14T23:15:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:15:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:15:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:15:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:15:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:15:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-14T23:15:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:15:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:15:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:15:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:15:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:15:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:15:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:15:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_40.status 2024-09-14T23:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_40.status 2024-09-14T23:15:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-14T23:15:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_40.status 2024-09-14T23:15:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.status 2024-09-14T23:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-14T23:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-14T23:15:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 164 2024-09-14T23:15:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_40.status 2024-09-14T23:15:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_40.status 2024-09-14T23:15:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-14T23:15:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:15:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=164 2024-09-14T23:15:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:15:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.status 2024-09-14T23:15:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.status 2024-09-14T23:15:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-14T23:15:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 164 = '' 2024-09-14T23:15:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.status 2024-09-14T23:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.status 2024-09-14T23:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-14T23:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 164 = '' 2024-09-14T23:15:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:15:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.status 2024-09-14T23:15:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.status 2024-09-14T23:15:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-14T23:15:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=164 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 164 = 164 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:15:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-14T23:15:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_40': 2024-09-14T23:15:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:15:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:15:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:15:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12a023087bd6 2024-09-14T23:15:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12a023087bd6 2024-09-14T23:15:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:15:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:21 2024 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:21 2024 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:21 2024 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 8837ff64-098e-4da2-a725-5197d788c4b5 2024-09-14T23:15:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-14T23:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_40.export 2024-09-14T23:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.export 2024-09-14T23:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_40.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.export 2024-09-14T23:15:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_40.export 2024-09-14T23:15:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.export 2024-09-14T23:15:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_40.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.export 2024-09-14T23:15:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_40.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_40.export 2024-09-14T23:15:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_41 2024-09-14T23:15:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-14T23:15:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:15:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:15:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:15:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:15:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:15:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:15:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-14T23:15:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-14T23:15:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-14T23:15:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:15:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:15:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:15:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:15:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:15:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:15:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:15:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-14T23:15:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:15:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:15:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:15:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:15:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:15:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-14T23:15:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_41.status 2024-09-14T23:15:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_41.status 2024-09-14T23:15:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-14T23:15:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_41.status 2024-09-14T23:15:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.status 2024-09-14T23:15:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-14T23:15:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-14T23:15:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 166 2024-09-14T23:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_41.status 2024-09-14T23:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_41.status 2024-09-14T23:15:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-14T23:15:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:15:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=166 2024-09-14T23:15:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:15:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.status 2024-09-14T23:15:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.status 2024-09-14T23:15:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-14T23:15:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 166 = '' 2024-09-14T23:15:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:15:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.status 2024-09-14T23:15:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.status 2024-09-14T23:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-14T23:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 166 = '' 2024-09-14T23:15:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:15:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.status 2024-09-14T23:15:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.status 2024-09-14T23:15:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-14T23:15:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=166 2024-09-14T23:15:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 166 = 166 2024-09-14T23:15:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-14T23:15:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:15:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-14T23:15:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_41': 2024-09-14T23:15:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:15:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:15:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:15:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e5551872fe 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e5551872fe 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:29 2024 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:29 2024 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:29 2024 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 56fdc800-4370-46f4-a73e-74ac75619223 2024-09-14T23:15:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:15:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-14T23:15:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:15:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:15:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_41.export 2024-09-14T23:15:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.export 2024-09-14T23:15:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_41.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.export 2024-09-14T23:15:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_41.export 2024-09-14T23:15:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.export 2024-09-14T23:15:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_41.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.export 2024-09-14T23:15:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_41.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_41.export 2024-09-14T23:15:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_42 2024-09-14T23:15:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-14T23:15:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-14T23:15:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:15:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:15:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-14T23:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-14T23:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:15:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:15:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:15:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:15:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-14T23:15:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-14T23:15:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-14T23:15:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:15:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:15:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:15:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:15:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-14T23:15:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-14T23:15:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:15:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:15:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:15:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2024-09-14T23:15:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-14T23:15:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:15:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:15:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_42 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:15:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:15:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:15:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:15:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:15:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_42 2024-09-14T23:15:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:15:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:15:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:15:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:15:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:15:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:15:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:15:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-14T23:15:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_42.status 2024-09-14T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_42.status 2024-09-14T23:15:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-14T23:15:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_42.status 2024-09-14T23:15:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.status 2024-09-14T23:15:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-14T23:15:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-14T23:15:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 168 2024-09-14T23:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_42.status 2024-09-14T23:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_42.status 2024-09-14T23:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-14T23:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:15:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=168 2024-09-14T23:15:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:15:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.status 2024-09-14T23:15:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.status 2024-09-14T23:15:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-14T23:15:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 168 = '' 2024-09-14T23:15:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:15:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.status 2024-09-14T23:15:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.status 2024-09-14T23:15:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-14T23:15:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 168 = '' 2024-09-14T23:15:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:15:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.status 2024-09-14T23:15:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.status 2024-09-14T23:15:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-14T23:15:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=168 2024-09-14T23:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 168 = 168 2024-09-14T23:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-14T23:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-14T23:15:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_42': 2024-09-14T23:15:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:15:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:15:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:15:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127efee50d65 2024-09-14T23:15:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127efee50d65 2024-09-14T23:15:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:15:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:15:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:15:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:15:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:07:39 2024 2024-09-14T23:15:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:07:39 2024 2024-09-14T23:15:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:07:39 2024 2024-09-14T23:15:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:15:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:15:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:15:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 1dbaf776-62e8-4913-8034-899563d552d8 2024-09-14T23:15:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:15:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-14T23:15:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:15:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:15:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_42.export 2024-09-14T23:15:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.export 2024-09-14T23:15:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_42.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.export 2024-09-14T23:15:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_42.export 2024-09-14T23:15:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.export 2024-09-14T23:15:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_42.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.export 2024-09-14T23:15:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_42.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_42.export 2024-09-14T23:15:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_43 2024-09-14T23:15:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-14T23:15:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-14T23:15:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:15:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:15:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:15:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-14T23:15:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:15:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:15:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:15:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:15:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:15:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:15:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:15:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:15:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:15:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:15:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:15:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:15:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:15:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:15:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:15:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:15:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:15:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:15:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:15:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:15:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-14T23:15:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:15:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:15:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:15:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:15:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:15:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:15:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:15:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-14T23:15:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:15:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:15:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-14T23:15:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:15:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-14T23:15:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_43.status 2024-09-14T23:15:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_43.status 2024-09-14T23:15:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-14T23:15:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_43.status 2024-09-14T23:15:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:15:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.status 2024-09-14T23:15:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-14T23:15:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-14T23:15:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 170 2024-09-14T23:15:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_43.status 2024-09-14T23:15:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_43.status 2024-09-14T23:15:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-14T23:15:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:15:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=170 2024-09-14T23:15:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:15:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.status 2024-09-14T23:15:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.status 2024-09-14T23:15:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-14T23:15:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 170 = '' 2024-09-14T23:15:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:15:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.status 2024-09-14T23:15:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.status 2024-09-14T23:15:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-14T23:15:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:15:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:15:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:15:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:15:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:15:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 170 = '' 2024-09-14T23:15:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:15:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:15:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.status 2024-09-14T23:15:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.status 2024-09-14T23:15:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-14T23:15:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:15:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:15:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=170 2024-09-14T23:15:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 170 = 170 2024-09-14T23:15:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-14T23:15:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:15:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:15:57.149 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.26440943932230054) > weight (0.25) 2024-09-14T23:15:57.149 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.38014166924243553) > weight (0.25) 2024-09-14T23:15:57.150 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-14T23:15:57.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-14T23:15:57.150 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-14T23:15:57.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:15:57.149+0000 7f167a5f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 38797) UID: 1000 2024-09-14T23:15:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-14T23:15:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_43': 2024-09-14T23:15:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:15:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:15:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:15:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e5aadab605 2024-09-14T23:15:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e5aadab605 2024-09-14T23:15:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:20 2024 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:20 2024 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:20 2024 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 800c56a4-4130-4ca1-a577-1c016136f961 2024-09-14T23:15:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:15:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-14T23:15:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:15:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:15:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_43.export 2024-09-14T23:15:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.export 2024-09-14T23:15:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_43.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.export 2024-09-14T23:15:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_43.export 2024-09-14T23:15:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.export 2024-09-14T23:15:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:15:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_43.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.export 2024-09-14T23:15:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_43.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_43.export 2024-09-14T23:15:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:15:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:15:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_44 2024-09-14T23:15:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-14T23:15:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:15:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:15:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:15:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:15:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:15:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-14T23:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:16:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-14T23:16:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:16:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:16:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:16:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:16:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:16:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:16:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:16:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-14T23:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:16:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:16:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:16:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:16:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-14T23:16:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:16:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:16:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:16:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:16:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:16:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:16:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:16:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_44.status 2024-09-14T23:16:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_44.status 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_44.status 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.status 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-14T23:16:01.151 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-14T23:16:01.151 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:16:01.151 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-14T23:16:01.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-14T23:16:01.152 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-14T23:16:01.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-14T23:16:01.155 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-09-14T23:16:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 172 2024-09-14T23:16:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_44.status 2024-09-14T23:16:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_44.status 2024-09-14T23:16:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-14T23:16:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:16:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=172 2024-09-14T23:16:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:16:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.status 2024-09-14T23:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.status 2024-09-14T23:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-14T23:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 172 = '' 2024-09-14T23:16:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:16:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.status 2024-09-14T23:16:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.status 2024-09-14T23:16:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-14T23:16:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 172 = '' 2024-09-14T23:16:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:16:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.status 2024-09-14T23:16:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.status 2024-09-14T23:16:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-14T23:16:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=172 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 172 = 172 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:16:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-14T23:16:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_44': 2024-09-14T23:16:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:16:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:16:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12a0dee2e52c 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12a0dee2e52c 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:00 2024 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:00 2024 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:00 2024 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 1657a08a-2ee0-4427-8918-43d3fca35d43 2024-09-14T23:16:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:16:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-14T23:16:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:16:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:16:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_44.export 2024-09-14T23:16:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.export 2024-09-14T23:16:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_44.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.export 2024-09-14T23:16:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_44.export 2024-09-14T23:16:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.export 2024-09-14T23:16:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_44.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.export 2024-09-14T23:16:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_44.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_44.export 2024-09-14T23:16:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_45 2024-09-14T23:16:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-14T23:16:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-14T23:16:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:16:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:16:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:16:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-14T23:16:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:16:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:16:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:16:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:16:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:16:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:16:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:16:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:16:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:16:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-14T23:16:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-14T23:16:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:16:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:16:06.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:16:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:16:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:16:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:16:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:16:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:16:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:16:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:16:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:16:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:16:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:16:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:16:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:16:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:16:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:16:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-14T23:16:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:16:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:16:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:16:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:16:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:16:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:16:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:16:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-14T23:16:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:16:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:16:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-14T23:16:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_45.status 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_45.status 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_45.status 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-14T23:16:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-14T23:16:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 174 2024-09-14T23:16:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_45.status 2024-09-14T23:16:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_45.status 2024-09-14T23:16:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-14T23:16:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:16:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=174 2024-09-14T23:16:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:16:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-14T23:16:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=79 2024-09-14T23:16:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 174 = 79 2024-09-14T23:16:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:16:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-14T23:16:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=79 2024-09-14T23:16:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 174 = 79 2024-09-14T23:16:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:16:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-14T23:16:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=79 2024-09-14T23:16:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 174 = 79 2024-09-14T23:16:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:16:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-14T23:16:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 174 = '' 2024-09-14T23:16:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:16:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.status 2024-09-14T23:16:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-14T23:16:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=174 2024-09-14T23:16:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 174 = 174 2024-09-14T23:16:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-14T23:16:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:16:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-14T23:16:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_45': 2024-09-14T23:16:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:16:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:16:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:16:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12aa8d7a84b0 2024-09-14T23:16:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12aa8d7a84b0 2024-09-14T23:16:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:16:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:16:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:16:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:16:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:15 2024 2024-09-14T23:16:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:15 2024 2024-09-14T23:16:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:15 2024 2024-09-14T23:16:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:16:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:16:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:16:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 384c1652-4afa-4fe7-8287-d17f60057ca1 2024-09-14T23:16:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_45.export 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.export 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_45.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.export 2024-09-14T23:16:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_45.export 2024-09-14T23:16:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.export 2024-09-14T23:16:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_45.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.export 2024-09-14T23:16:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_45.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_45.export 2024-09-14T23:16:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_46 2024-09-14T23:16:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-14T23:16:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:16:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-14T23:16:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:16:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:16:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:16:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-14T23:16:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:16:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-14T23:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:16:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:16:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:16:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-14T23:16:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-14T23:16:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:16:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:16:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-14T23:16:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:16:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:16:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:16:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:16:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:16:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:16:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:16:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:16:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-14T23:16:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-14T23:16:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:16:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:16:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:16:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:16:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:16:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:16:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-14T23:16:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-14T23:16:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:16:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:3 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:3 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=3 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=3 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:3 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:3 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=3 2024-09-14T23:16:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T23:16:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=3 2024-09-14T23:16:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=3 2024-09-14T23:16:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T23:16:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:16:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:16:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T23:16:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2024-09-14T23:16:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:16:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:16:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:16:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:16:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:16:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:16:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:16:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-14T23:16:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_46.status 2024-09-14T23:16:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_46.status 2024-09-14T23:16:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-14T23:16:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_46.status 2024-09-14T23:16:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.status 2024-09-14T23:16:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-14T23:16:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-14T23:16:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 176 2024-09-14T23:16:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_46.status 2024-09-14T23:16:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_46.status 2024-09-14T23:16:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-14T23:16:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:16:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=176 2024-09-14T23:16:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.status 2024-09-14T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.status 2024-09-14T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-14T23:16:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 176 = '' 2024-09-14T23:16:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:16:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.status 2024-09-14T23:16:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.status 2024-09-14T23:16:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-14T23:16:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 176 = '' 2024-09-14T23:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:16:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.status 2024-09-14T23:16:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.status 2024-09-14T23:16:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-14T23:16:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=176 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 176 = 176 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:16:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-14T23:16:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_46': 2024-09-14T23:16:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:16:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127edac04bcd 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127edac04bcd 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:26 2024 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:26 2024 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:26 2024 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:16:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:16:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:16:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: fbfee603-c1c3-48f3-b0df-17ec0d3552e7 2024-09-14T23:16:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:16:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-14T23:16:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:16:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:16:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_46.export 2024-09-14T23:16:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.export 2024-09-14T23:16:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_46.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.export 2024-09-14T23:16:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_46.export 2024-09-14T23:16:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.export 2024-09-14T23:16:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_46.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.export 2024-09-14T23:16:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_46.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_46.export 2024-09-14T23:16:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_47 2024-09-14T23:16:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-14T23:16:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:16:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-14T23:16:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:16:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:16:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:16:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-14T23:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:16:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:16:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:16:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:16:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-14T23:16:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:16:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:16:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:16:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:16:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:16:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:16:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-14T23:16:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_47.status 2024-09-14T23:16:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_47.status 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_47.status 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.status 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-14T23:16:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 178 2024-09-14T23:16:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_47.status 2024-09-14T23:16:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_47.status 2024-09-14T23:16:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-14T23:16:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=178 2024-09-14T23:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:16:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.status 2024-09-14T23:16:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.status 2024-09-14T23:16:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-14T23:16:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 178 = '' 2024-09-14T23:16:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:16:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.status 2024-09-14T23:16:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.status 2024-09-14T23:16:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-14T23:16:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 178 = '' 2024-09-14T23:16:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:16:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.status 2024-09-14T23:16:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.status 2024-09-14T23:16:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-14T23:16:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=178 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 178 = 178 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-14T23:16:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_47': 2024-09-14T23:16:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12a0f672f6cf 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12a0f672f6cf 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:36 2024 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:36 2024 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:36 2024 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:16:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:16:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:16:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 9dc01f72-5b4f-4251-988c-6e2524bebd28 2024-09-14T23:16:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:16:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-14T23:16:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:16:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:16:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_47.export 2024-09-14T23:16:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.export 2024-09-14T23:16:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_47.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.export 2024-09-14T23:16:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_47.export 2024-09-14T23:16:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.export 2024-09-14T23:16:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_47.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.export 2024-09-14T23:16:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_47.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_47.export 2024-09-14T23:16:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_48 2024-09-14T23:16:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-14T23:16:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:16:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-14T23:16:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:16:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:16:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-14T23:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:16:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-14T23:16:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:16:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:16:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:16:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:16:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:16:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:16:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-14T23:16:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_48.status 2024-09-14T23:16:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_48.status 2024-09-14T23:16:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-14T23:16:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_48.status 2024-09-14T23:16:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.status 2024-09-14T23:16:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-14T23:16:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-14T23:16:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 180 2024-09-14T23:16:28.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_48.status 2024-09-14T23:16:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_48.status 2024-09-14T23:16:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-14T23:16:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:16:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=180 2024-09-14T23:16:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:16:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.status 2024-09-14T23:16:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.status 2024-09-14T23:16:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-14T23:16:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=82 2024-09-14T23:16:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 180 = 82 2024-09-14T23:16:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:16:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.status 2024-09-14T23:16:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.status 2024-09-14T23:16:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-14T23:16:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 180 = '' 2024-09-14T23:16:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:16:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.status 2024-09-14T23:16:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.status 2024-09-14T23:16:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-14T23:16:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 180 = '' 2024-09-14T23:16:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:16:31.161 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7028032962196117) > weight (0.25) 2024-09-14T23:16:31.162 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3599161266327403) > weight (0.25) 2024-09-14T23:16:31.162 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4316346283826804) > weight (0.25) 2024-09-14T23:16:31.162 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4998180816435187) > weight (0.25) 2024-09-14T23:16:31.162 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-14T23:16:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.status 2024-09-14T23:16:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.status 2024-09-14T23:16:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-14T23:16:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=180 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 180 = 180 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:16:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-14T23:16:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_48': 2024-09-14T23:16:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:16:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:16:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:16:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 12aaebe721f2 2024-09-14T23:16:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.12aaebe721f2 2024-09-14T23:16:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:08:49 2024 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:08:49 2024 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:08:49 2024 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 11b0a986-da1c-46e1-bd44-b5b3d1b231b5 2024-09-14T23:16:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:16:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-14T23:16:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:16:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:16:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_48.export 2024-09-14T23:16:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.export 2024-09-14T23:16:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_48.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.export 2024-09-14T23:16:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_48.export 2024-09-14T23:16:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.export 2024-09-14T23:16:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_48.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.export 2024-09-14T23:16:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_48.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_48.export 2024-09-14T23:16:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_49 2024-09-14T23:16:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-14T23:16:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:16:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-14T23:16:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:16:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:16:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:16:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-14T23:16:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:16:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:16:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:16:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:16:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:16:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:16:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:16:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:16:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-14T23:16:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:16:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:16:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:16:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:16:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:16:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:16:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-14T23:16:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:16:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:16:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:16:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:16:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:16:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-14T23:16:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_49.status 2024-09-14T23:16:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_49.status 2024-09-14T23:16:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-14T23:16:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_49.status 2024-09-14T23:16:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.status 2024-09-14T23:16:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-14T23:16:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-14T23:16:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 182 2024-09-14T23:16:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_49.status 2024-09-14T23:16:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_49.status 2024-09-14T23:16:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-14T23:16:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:16:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=182 2024-09-14T23:16:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:16:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.status 2024-09-14T23:16:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.status 2024-09-14T23:16:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-14T23:16:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=83 2024-09-14T23:16:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 182 = 83 2024-09-14T23:16:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:16:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.status 2024-09-14T23:16:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.status 2024-09-14T23:16:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-14T23:16:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=83 2024-09-14T23:16:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 182 = 83 2024-09-14T23:16:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:16:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.status 2024-09-14T23:16:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.status 2024-09-14T23:16:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-14T23:16:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 182 = '' 2024-09-14T23:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:16:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.status 2024-09-14T23:16:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.status 2024-09-14T23:16:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-14T23:16:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=182 2024-09-14T23:16:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 182 = 182 2024-09-14T23:16:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-14T23:16:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:16:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-14T23:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_49': 2024-09-14T23:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 10e53015baa1 2024-09-14T23:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.10e53015baa1 2024-09-14T23:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:09:03 2024 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:09:03 2024 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:09:03 2024 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 909132f1-ded4-4f35-97e5-a59b71439534 2024-09-14T23:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-14T23:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_49.export 2024-09-14T23:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.export 2024-09-14T23:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_49.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.export 2024-09-14T23:16:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_49.export 2024-09-14T23:16:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.export 2024-09-14T23:16:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_49.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.export 2024-09-14T23:16:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_49.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_49.export 2024-09-14T23:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_50 2024-09-14T23:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-14T23:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T23:16:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-14T23:16:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=started 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-14T23:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=started 2024-09-14T23:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_result 2024-09-14T23:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=stopped 2024-09-14T23:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-14T23:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_instance=cluster1 2024-09-14T23:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -i state 2024-09-14T23:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance=cluster1 2024-09-14T23:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local loop_instance 2024-09-14T23:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T23:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sleep 0 2024-09-14T23:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T23:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ seq 0 3 2024-09-14T23:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-14T23:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:0 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:0 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=0 2024-09-14T23:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:0 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:0 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=0 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=0 2024-09-14T23:16:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=0 2024-09-14T23:16:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T23:16:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T23:16:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-14T23:16:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-14T23:16:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:16:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:1 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:1 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=1 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T23:16:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=1 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:1 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:1 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=1 2024-09-14T23:16:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T23:16:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=1 2024-09-14T23:16:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=1 2024-09-14T23:16:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T23:16:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:16:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:16:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T23:16:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-14T23:16:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-14T23:16:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T23:16:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:admin_socket: invalid command 2024-09-14T23:16:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:16:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-09-14T23:16:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1:2 2024-09-14T23:16:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ shift 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local val=cluster1:2 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster_var_name=cluster 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local instance_var_name=instance 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=2 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval cluster=cluster1 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ eval instance=2 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T23:16:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local local_cluster=cluster1:2 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster=cluster1 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local val=cluster1:2 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ local instance_var_name=instance 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ cluster=cluster1 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ instance=2 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval local_cluster=cluster1 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ local_cluster=cluster1 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ eval instance=2 2024-09-14T23:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ instance=2 2024-09-14T23:16:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T23:16:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:16:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:16:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T23:16:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-09-14T23:16:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ return 0 2024-09-14T23:16:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T23:16:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo ' "state": "Replaying"' 2024-09-14T23:16:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ grep -i Replaying 2024-09-14T23:16:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: "state": "Replaying" 2024-09-14T23:16:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=started 2024-09-14T23:16:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test started = started 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = journal ']' 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_cluster=cluster1 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-14T23:16:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster2-mirror-image_50.status 2024-09-14T23:16:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local status_log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_50.status 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ echo cluster1-mirror-image_50.status 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local local_status_log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.status 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-14T23:16:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Snapshot ID: 184 2024-09-14T23:16:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_50.status 2024-09-14T23:16:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_50.status 2024-09-14T23:16:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-14T23:16:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T23:16:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snapshot_id=184 2024-09-14T23:16:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.2 2024-09-14T23:16:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.status 2024-09-14T23:16:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.status 2024-09-14T23:16:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-14T23:16:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=84 2024-09-14T23:16:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 184 = 84 2024-09-14T23:16:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.4 2024-09-14T23:16:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.status 2024-09-14T23:16:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.status 2024-09-14T23:16:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-14T23:16:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=84 2024-09-14T23:16:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 184 = 84 2024-09-14T23:16:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.8 2024-09-14T23:16:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.status 2024-09-14T23:16:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.status 2024-09-14T23:16:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-14T23:16:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:16:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:-:1.1: Document is empty 2024-09-14T23:16:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: 2024-09-14T23:16:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:^ 2024-09-14T23:16:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id= 2024-09-14T23:16:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 184 = '' 2024-09-14T23:16:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T23:16:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1.6 2024-09-14T23:16:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.status 2024-09-14T23:16:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local log=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.status 2024-09-14T23:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-14T23:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T23:16:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local primary_snapshot_id=184 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 184 = 184 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T23:16:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:16:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-14T23:16:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_50': 2024-09-14T23:16:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 128 MiB in 32 objects 2024-09-14T23:16:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:16:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 2 2024-09-14T23:16:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 127e4ab6b8f6 2024-09-14T23:16:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.127e4ab6b8f6 2024-09-14T23:16:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:16:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering, non-primary 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:09:12 2024 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:09:12 2024 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:09:12 2024 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: protected: False 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: af809fea-94db-4aaf-b996-c34cbeff8227 2024-09-14T23:16:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: false 2024-09-14T23:16:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-14T23:16:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:16:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap_name=snap 2024-09-14T23:16:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local ret=0 2024-09-14T23:16:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local rmt_export=/tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_50.export 2024-09-14T23:16:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local loc_export=/tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.export 2024-09-14T23:16:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_50.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.export 2024-09-14T23:16:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_50.export 2024-09-14T23:16:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.export 2024-09-14T23:16:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T23:16:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cmp /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_50.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.export 2024-09-14T23:16:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -f /tmp/tmp.OxLGv2e3xU/cluster2-mirror-image_50.export /tmp/tmp.OxLGv2e3xU/cluster1-mirror-image_50.export 2024-09-14T23:16:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ testlog 'TEST: delete many images' 2024-09-14T23:16:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ tee -a /tmp/tmp.OxLGv2e3xU/rbd-mirror.test.log 2024-09-14T23:16:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ date '+%F %T' 2024-09-14T23:16:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo 2024-09-14 23:16:49 TEST: delete many images 2024-09-14T23:16:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:2024-09-14 23:16:49 TEST: delete many images 2024-09-14T23:16:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 50 2024-09-14T23:16:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_1 2024-09-14T23:16:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-14T23:16:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:16:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:16:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-14T23:16:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:16:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-14T23:16:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:16:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:16:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:16:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T23:16:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:16:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T23:16:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T23:16:49.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:16:49.652+0000 7fdde704d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23e8a9040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:16:49.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:16:49.652+0000 7fdded85a640 -1 rbd::mirror::ImageReplayer: 0x55a23ede3400 [3/23baaf62-5401-42f0-ae98-c17bcc416c67] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:16:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:16:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_2 2024-09-14T23:16:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-14T23:16:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:16:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:16:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-14T23:16:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:16:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-14T23:16:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:16:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:16:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:16:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T23:16:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:16:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T23:16:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T23:16:51.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:16:51.678+0000 7fdde704d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23e8a8ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:16:51.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:16:51.678+0000 7fdded059640 -1 rbd::mirror::ImageReplayer: 0x55a23ede2500 [3/13ef0e31-a893-45b8-a2c7-d21116cdc208] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:16:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:16:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_3 2024-09-14T23:16:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-14T23:16:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:16:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:16:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-14T23:16:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:16:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-14T23:16:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:16:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:16:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:16:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T23:16:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:16:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T23:16:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T23:16:53.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:16:53.670+0000 7fdde704d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23e68c9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:16:53.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:16:53.670+0000 7fdded059640 -1 rbd::mirror::ImageReplayer: 0x55a23ede3680 [3/2903cd12-fe5f-4679-a7de-9ede74f67058] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:16:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:16:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_4 2024-09-14T23:16:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-14T23:16:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:16:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:16:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-14T23:16:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:16:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-14T23:16:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:16:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:16:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:16:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T23:16:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:16:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T23:16:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T23:16:55.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:16:55.687+0000 7fd6073db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e1520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:16:55.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:16:55.687+0000 7fd60dbe8640 -1 rbd::mirror::ImageReplayer: 0x55ed7cfb2000 [3/d53f0a91-12f4-4ba8-846e-d09778cdee6d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:16:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:16:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_5 2024-09-14T23:16:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-14T23:16:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:16:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:16:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-14T23:16:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:16:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-14T23:16:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:16:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:16:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:16:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T23:16:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:16:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T23:16:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T23:16:57.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:16:57.751+0000 7f1fb0337640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b109e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:16:57.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:16:57.751+0000 7f1fb6343640 -1 rbd::mirror::ImageReplayer: 0x5603b182bb80 [3/cd71c6af-1235-405d-96be-3b0fea37da92] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:16:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:16:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:16:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:16:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_6 2024-09-14T23:16:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-14T23:16:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:16:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:16:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-14T23:16:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:16:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-14T23:16:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:16:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:16:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:16:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T23:16:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:16:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:16:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:16:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T23:16:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T23:16:59.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:16:59.754+0000 7fdde704d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23e8a9380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:16:59.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:16:59.754+0000 7fdded85a640 -1 rbd::mirror::ImageReplayer: 0x55a23ee68000 [3/63095e72-2322-434d-9a80-ed8a5dcddbff] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_7 2024-09-14T23:17:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-14T23:17:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:17:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-14T23:17:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-14T23:17:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:17:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T23:17:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:17:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T23:17:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T23:17:01.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:01.743+0000 7fd6073db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e1380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:01.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:01.743+0000 7fd60dbe8640 -1 rbd::mirror::ImageReplayer: 0x55ed7cf4d900 [3/b412e86d-c0d6-4c66-9336-1d3894abecba] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_8 2024-09-14T23:17:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-14T23:17:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:17:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-14T23:17:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-14T23:17:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:17:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T23:17:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:17:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T23:17:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T23:17:03.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:17:03.783+0000 7f1fafb36640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b109a820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:03.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:17:03.783+0000 7f1fb6b44640 -1 rbd::mirror::ImageReplayer: 0x5603b182a000 [3/1183b733-6e8d-4fa0-bcc5-8cdba88bdd37] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_9 2024-09-14T23:17:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-14T23:17:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:17:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-14T23:17:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-14T23:17:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:17:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T23:17:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:17:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T23:17:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T23:17:05.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:05.786+0000 7fd6073db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e11e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:05.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:05.786+0000 7fd60dbe8640 -1 rbd::mirror::ImageReplayer: 0x55ed7cfb2280 [3/e951d895-861e-422c-881e-e2fec0aaeafd] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_10 2024-09-14T23:17:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-14T23:17:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:17:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-14T23:17:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-14T23:17:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:17:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T23:17:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:17:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T23:17:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T23:17:07.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:07.790+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfbcb7ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:07.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:07.790+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc50ac80 [3/fd912f05-09a2-4d03-a711-51d6bfba6b89] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_11 2024-09-14T23:17:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-14T23:17:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:17:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-14T23:17:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-14T23:17:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:17:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_11 2024-09-14T23:17:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:17:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-14T23:17:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-14T23:17:09.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:09.805+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfc417a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:09.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:09.805+0000 7fb3e21e7640 -1 rbd::mirror::ImageReplayer: 0x558cfc415680 [3/3d92657a-d871-4f0e-87e5-49062e513296] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_12 2024-09-14T23:17:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-14T23:17:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:17:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-14T23:17:11.547 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T23:17:11.545+0000 7f5706106640 -1 reset not still connected to 0x5560e697f790 2024-09-14T23:17:11.547 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T23:17:11.545+0000 7f5706106640 -1 reset not still connected to 0x5560e697fa00 2024-09-14T23:17:11.548 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T23:17:11.545+0000 7f5706106640 -1 reset not still connected to 0x5560e697fba0 2024-09-14T23:17:11.548 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T23:17:11.545+0000 7f5706106640 -1 reset not still connected to 0x5560e7d6fad0 2024-09-14T23:17:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-14T23:17:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:17:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_12 2024-09-14T23:17:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:17:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-14T23:17:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-14T23:17:11.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:11.749+0000 7fd6073db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e1040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:11.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:11.749+0000 7fd60dbe8640 -1 rbd::mirror::ImageReplayer: 0x55ed7cfb2500 [3/64ee2f3d-5dac-4606-aaf0-b005a4416265] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_13 2024-09-14T23:17:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-14T23:17:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:17:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-14T23:17:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-14T23:17:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:17:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_13 2024-09-14T23:17:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:17:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-14T23:17:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-14T23:17:13.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:13.822+0000 7fd6073db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:13.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:13.822+0000 7fd60d3e7640 -1 rbd::mirror::ImageReplayer: 0x55ed7cf4db80 [3/ca7825af-19b6-473a-a8e5-328e157a8d1a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:14.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_14 2024-09-14T23:17:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-14T23:17:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:17:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-14T23:17:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-14T23:17:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:17:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_14 2024-09-14T23:17:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:17:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-14T23:17:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-14T23:17:15.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:15.818+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfbbc64e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:15.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:15.818+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc415900 [3/45c698d2-2f4b-4209-84b4-d26cd2e23680] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_15 2024-09-14T23:17:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-14T23:17:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:17:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-14T23:17:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_15 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:17:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-14T23:17:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-14T23:17:17.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:17.827+0000 7fb3dc1db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfc416ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:17.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:17.827+0000 7fb3e21e7640 -1 rbd::mirror::ImageReplayer: 0x558cfc414280 [3/77c7ed00-9d21-46b0-b7ed-332bbd914227] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_16 2024-09-14T23:17:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-14T23:17:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:17:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-14T23:17:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-14T23:17:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:17:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_16 2024-09-14T23:17:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:17:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-14T23:17:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-14T23:17:19.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:19.827+0000 7fdde704d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23e58da00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_17 2024-09-14T23:17:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-14T23:17:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:17:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-14T23:17:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-14T23:17:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:17:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_17 2024-09-14T23:17:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:17:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-14T23:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-14T23:17:21.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:21.743+0000 7fd606bda640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e1040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:21.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:21.743+0000 7fd60d3e7640 -1 rbd::mirror::ImageReplayer: 0x55ed7cf4cc80 [3/839cf198-206a-4346-91c9-f4360063f32f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_18 2024-09-14T23:17:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-14T23:17:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:17:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-14T23:17:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_18 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:17:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-14T23:17:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-14T23:17:23.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:23.844+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfc417a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:23.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:23.844+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc414780 [3/d80a064e-80a3-476f-9ce2-c2191cc27f0b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_19 2024-09-14T23:17:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-14T23:17:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:17:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-14T23:17:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-14T23:17:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:17:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_19 2024-09-14T23:17:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:17:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-14T23:17:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-14T23:17:25.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:25.858+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfc416b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:25.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:25.858+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc414000 [3/5b5b353a-9976-4517-9150-396dd4235407] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_20 2024-09-14T23:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-14T23:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-14T23:17:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-14T23:17:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:17:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_20 2024-09-14T23:17:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:17:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-14T23:17:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-14T23:17:27.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:27.856+0000 7fb3dc1db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfc4161a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:27.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:27.856+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc415400 [3/c471720c-4eb1-4c8a-a244-9687727d167b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_21 2024-09-14T23:17:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-14T23:17:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:17:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-14T23:17:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-14T23:17:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:17:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_21 2024-09-14T23:17:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:17:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-14T23:17:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-14T23:17:29.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:17:29.860+0000 7f1fb0337640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b109fa00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_22 2024-09-14T23:17:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-14T23:17:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:17:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-14T23:17:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-14T23:17:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:17:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_22 2024-09-14T23:17:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:17:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-14T23:17:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-14T23:17:31.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:31.752+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfc417040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:31.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:31.752+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc414500 [3/a9dd159e-4591-405f-b0f6-42ca7c8cdf26] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_23 2024-09-14T23:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-14T23:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-14T23:17:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-14T23:17:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:17:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_23 2024-09-14T23:17:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:17:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-14T23:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-14T23:17:33.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:33.883+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfc417380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:33.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:33.883+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc414a00 [3/f42fac99-ab54-44f2-ac18-c845cd99d02b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_24 2024-09-14T23:17:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-14T23:17:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:17:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-14T23:17:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-14T23:17:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:17:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_24 2024-09-14T23:17:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:17:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-14T23:17:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-14T23:17:36.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:36.087+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfbede680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:36.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:36.087+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc50a000 [3/d02fb2d0-2bf7-49a8-86d4-ecd3c3240c86] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:36.677 INFO:tasks.ceph.cluster2.osd.1.smithi089.stderr:2024-09-14T23:17:36.675+0000 7fdab2495640 -1 reset not still connected to 0x55effcd3ba00 2024-09-14T23:17:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_25 2024-09-14T23:17:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-14T23:17:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:17:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-14T23:17:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-14T23:17:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:17:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_25 2024-09-14T23:17:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:17:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-14T23:17:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-14T23:17:37.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:37.904+0000 7fd6073db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:37.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:37.904+0000 7fd60d3e7640 -1 rbd::mirror::ImageReplayer: 0x55ed7cf4c280 [3/16ceb1c6-1a5c-4b7b-8b20-da6200f477b6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_26 2024-09-14T23:17:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-14T23:17:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:17:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-14T23:17:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-14T23:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:17:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_26 2024-09-14T23:17:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:17:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-14T23:17:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-14T23:17:39.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:17:39.906+0000 7f1fafb36640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b0fa9380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:39.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:17:39.906+0000 7f1fb6b44640 -1 rbd::mirror::ImageReplayer: 0x5603b182b680 [3/68d375d5-545a-4f5c-87a8-2ef5ae048705] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_27 2024-09-14T23:17:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-14T23:17:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:17:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-14T23:17:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-14T23:17:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:17:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_27 2024-09-14T23:17:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:17:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-14T23:17:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-14T23:17:41.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:41.744+0000 7fdde704d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23e68c9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:41.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:41.744+0000 7fdded059640 -1 rbd::mirror::ImageReplayer: 0x55a23ede2280 [3/a848ce88-5df8-4b90-b6a1-72b3261b72bc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_28 2024-09-14T23:17:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-14T23:17:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:17:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-14T23:17:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-14T23:17:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:17:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_28 2024-09-14T23:17:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:17:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-14T23:17:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-14T23:17:43.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:17:43.777+0000 7f1fb0337640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b0fa91e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:43.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:17:43.777+0000 7f1fb6b44640 -1 rbd::mirror::ImageReplayer: 0x5603b182af00 [3/4c9c9805-ba1a-48da-a08b-8de70de557b3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_29 2024-09-14T23:17:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-14T23:17:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:17:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-14T23:17:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-14T23:17:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:17:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_29 2024-09-14T23:17:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:17:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-14T23:17:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-14T23:17:45.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:45.753+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfbbc71e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:45.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:45.753+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc414f00 [3/ad5acb85-55a5-4fcf-abcf-6bf6f2766d18] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_30 2024-09-14T23:17:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-14T23:17:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:17:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-14T23:17:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-14T23:17:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:17:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_30 2024-09-14T23:17:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:17:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-14T23:17:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-14T23:17:47.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:47.764+0000 7fdde684c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23e58d380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:47.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:47.764+0000 7fdded059640 -1 rbd::mirror::ImageReplayer: 0x55a23ede2000 [3/2dd3a48d-e1ef-4b2b-b6fb-173793741ecc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_31 2024-09-14T23:17:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-14T23:17:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:17:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-14T23:17:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-14T23:17:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:17:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_31 2024-09-14T23:17:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:17:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-14T23:17:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-14T23:17:49.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:49.782+0000 7fdde684c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23e8aa9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:49.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:49.782+0000 7fdded059640 -1 rbd::mirror::ImageReplayer: 0x55a23ede2780 [3/d5f8f61a-4276-4546-8e0b-db256cf1d59a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_32 2024-09-14T23:17:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-14T23:17:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:17:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-14T23:17:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-14T23:17:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:17:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_32 2024-09-14T23:17:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-14T23:17:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-14T23:17:51.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:51.948+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfbbc76c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:51.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:17:51.948+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc415b80 [3/79596924-7c99-4578-ab4b-9b96094cd7b2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_33 2024-09-14T23:17:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-14T23:17:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:17:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-14T23:17:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-14T23:17:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:17:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_33 2024-09-14T23:17:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:17:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-14T23:17:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-14T23:17:53.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:53.786+0000 7fdde704d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23e688d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:53.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:53.786+0000 7fdded85a640 -1 rbd::mirror::ImageReplayer: 0x55a23ede3900 [3/4c528d36-e484-45fe-9609-82252ec0ca3f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:54.156 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T23:17:54.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T23:17:54.157 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T23:17:54.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:17:54.157+0000 7f4835f30640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 59075) UID: 1000 2024-09-14T23:17:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_34 2024-09-14T23:17:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-14T23:17:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:17:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-14T23:17:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:55.652 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T23:17:55.651+0000 7f5706106640 -1 reset not still connected to 0x5560e697fa00 2024-09-14T23:17:55.652 INFO:tasks.ceph.cluster1.osd.0.smithi052.stderr:2024-09-14T23:17:55.651+0000 7f5706106640 -1 reset not still connected to 0x5560e697fba0 2024-09-14T23:17:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-14T23:17:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:17:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_34 2024-09-14T23:17:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:17:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-14T23:17:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-14T23:17:55.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:55.799+0000 7fd6073db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e1520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:55.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:55.799+0000 7fd60d3e7640 -1 rbd::mirror::ImageReplayer: 0x55ed7cf4d680 [3/b3b49072-90fc-4a20-87f1-ba0c9f24cea9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_35 2024-09-14T23:17:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-14T23:17:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:17:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-14T23:17:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-14T23:17:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:17:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_35 2024-09-14T23:17:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:17:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-14T23:17:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-14T23:17:57.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:57.813+0000 7fdde684c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23f3ec820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:57.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:17:57.813+0000 7fdded85a640 -1 rbd::mirror::ImageReplayer: 0x55a23ede3b80 [3/62edd50a-04e9-49e3-95e0-0652ce1bea79] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:17:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:17:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:17:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:17:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_36 2024-09-14T23:17:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-14T23:17:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:17:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:17:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-14T23:17:59.158 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T23:17:59.158 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:17:59.159 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T23:17:59.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T23:17:59.159 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T23:17:59.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T23:17:59.162 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-09-14T23:17:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:17:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-14T23:17:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:17:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:17:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:17:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_36 2024-09-14T23:17:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:17:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:17:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:17:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-14T23:17:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-14T23:17:59.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:59.807+0000 7fd606bda640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e1380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:17:59.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:17:59.807+0000 7fd60d3e7640 -1 rbd::mirror::ImageReplayer: 0x55ed7cf4cf00 [3/9def2b16-9e96-4962-a4ee-3a7a75bafdd6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_37 2024-09-14T23:18:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-14T23:18:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:18:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-14T23:18:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-14T23:18:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:18:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_37 2024-09-14T23:18:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:18:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-14T23:18:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-14T23:18:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_38 2024-09-14T23:18:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-14T23:18:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:18:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-14T23:18:03.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-14T23:18:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:18:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_38 2024-09-14T23:18:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:18:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-14T23:18:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-14T23:18:03.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:03.841+0000 7f1fb0337640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b109f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:18:03.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:03.841+0000 7f1fb6343640 -1 rbd::mirror::ImageReplayer: 0x5603b182ac80 [3/3f0fd191-d8d6-479a-bc37-05f42708eaf7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_39 2024-09-14T23:18:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-14T23:18:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:18:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-14T23:18:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-14T23:18:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:18:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_39 2024-09-14T23:18:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:18:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-14T23:18:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-14T23:18:05.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:05.850+0000 7f1fb0337640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b0fa9860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:18:05.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:05.850+0000 7f1fb6b44640 -1 rbd::mirror::ImageReplayer: 0x5603b182aa00 [3/2c75e949-74f5-488e-aeb1-ce0ee7bef852] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_40 2024-09-14T23:18:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-14T23:18:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:18:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-14T23:18:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-14T23:18:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:18:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_40 2024-09-14T23:18:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:18:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-14T23:18:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-14T23:18:07.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:18:07.806+0000 7ff8152f7640 -1 rbd::mirror::ImageMap: 0x5573dcbf0000 handle_peer_ack_remove: failed to remove global_image_id=8837ff64-098e-4da2-a725-5197d788c4b5 2024-09-14T23:18:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_41 2024-09-14T23:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-14T23:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-14T23:18:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-14T23:18:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:18:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_41 2024-09-14T23:18:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:18:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-14T23:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-14T23:18:09.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:18:09.837+0000 7f39b1344640 -1 rbd::mirror::ImageReplayer: 0x55d65b0ee000 [3/56fdc800-4370-46f4-a73e-74ac75619223] start: already running 2024-09-14T23:18:09.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:18:09.865+0000 7fd606bda640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c9e0680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:18:09.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:18:09.865+0000 7fd60d3e7640 -1 rbd::mirror::ImageReplayer: 0x55ed7cf4c780 [3/56fdc800-4370-46f4-a73e-74ac75619223] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_42 2024-09-14T23:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-14T23:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-14T23:18:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-14T23:18:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:18:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_42 2024-09-14T23:18:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:18:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-14T23:18:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-14T23:18:11.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:11.759+0000 7f1fb0337640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b12c7ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:18:11.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:11.759+0000 7f1fb6343640 -1 rbd::mirror::ImageReplayer: 0x5603b182a780 [3/1dbaf776-62e8-4913-8034-899563d552d8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_43 2024-09-14T23:18:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-14T23:18:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:18:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-14T23:18:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-14T23:18:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:18:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_43 2024-09-14T23:18:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:18:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-14T23:18:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-14T23:18:13.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:18:13.822+0000 7ff8152f7640 -1 rbd::mirror::ImageMap: 0x5573dcbf0000 handle_peer_ack_remove: failed to remove global_image_id=800c56a4-4130-4ca1-a577-1c016136f961 2024-09-14T23:18:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_44 2024-09-14T23:18:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-14T23:18:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:18:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-14T23:18:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-14T23:18:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:18:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_44 2024-09-14T23:18:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:18:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-14T23:18:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-14T23:18:15.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:18:15.882+0000 7ff8152f7640 -1 rbd::mirror::ImageMap: 0x5573dcbf0000 handle_peer_ack_remove: failed to remove global_image_id=1657a08a-2ee0-4427-8918-43d3fca35d43 2024-09-14T23:18:15.891 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T23:18:15.890+0000 7f5fe0b61640 -1 reset not still connected to 0x55926a727380 2024-09-14T23:18:15.892 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T23:18:15.891+0000 7f5fe0b61640 -1 reset not still connected to 0x5592735851e0 2024-09-14T23:18:15.892 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T23:18:15.891+0000 7f5fe0b61640 -1 reset not still connected to 0x55927543b450 2024-09-14T23:18:15.892 INFO:tasks.ceph.cluster2.osd.2.smithi089.stderr:2024-09-14T23:18:15.891+0000 7f5fe0b61640 -1 reset not still connected to 0x559284a449c0 2024-09-14T23:18:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_45 2024-09-14T23:18:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-14T23:18:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:18:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-14T23:18:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-14T23:18:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:18:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_45 2024-09-14T23:18:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:18:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-14T23:18:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-14T23:18:17.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:17.888+0000 7f1fafb36640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b12c6d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:18:17.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:17.888+0000 7f1fb6343640 -1 rbd::mirror::ImageReplayer: 0x5603b182a500 [3/384c1652-4afa-4fe7-8287-d17f60057ca1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_46 2024-09-14T23:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-14T23:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-14T23:18:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-14T23:18:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:18:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_46 2024-09-14T23:18:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:18:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-14T23:18:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-14T23:18:19.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:18:19.900+0000 7fdde684c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a23f3ecb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:18:19.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:18:19.900+0000 7fdded85a640 -1 rbd::mirror::ImageReplayer: 0x55a23f2cc000 [3/fbfee603-c1c3-48f3-b0df-17ec0d3552e7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_47 2024-09-14T23:18:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-14T23:18:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:18:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-14T23:18:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-14T23:18:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:18:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_47 2024-09-14T23:18:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:18:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-14T23:18:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-14T23:18:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_48 2024-09-14T23:18:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-14T23:18:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:18:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-14T23:18:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-14T23:18:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:18:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_48 2024-09-14T23:18:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:18:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-14T23:18:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-14T23:18:23.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:23.754+0000 7f1fafb36640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b12c4d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:18:23.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:23.754+0000 7f1fb6b44640 -1 rbd::mirror::ImageReplayer: 0x5603b182a280 [3/11b0a986-da1c-46e1-bd44-b5b3d1b231b5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_49 2024-09-14T23:18:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-14T23:18:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:18:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-14T23:18:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-14T23:18:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:18:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_49 2024-09-14T23:18:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:18:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-14T23:18:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-14T23:18:25.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:18:25.963+0000 7fd6073db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ed7c7c0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:18:25.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:18:25.963+0000 7fd60dbe8640 -1 rbd::mirror::ImageReplayer: 0x55ed7cf4c500 [3/909132f1-ded4-4f35-97e5-a59b71439534] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_50 2024-09-14T23:18:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-14T23:18:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:18:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local snap=snap 2024-09-14T23:18:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-14T23:18:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing snap: 100% complete...done. 2024-09-14T23:18:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-14T23:18:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:18:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_50 2024-09-14T23:18:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:18:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-14T23:18:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-14T23:18:27.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:27.769+0000 7f1fafb36640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b0fa91e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:18:27.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:18:27.769+0000 7f1fb6b44640 -1 rbd::mirror::ImageReplayer: 0x5603b182b900 [3/af809fea-94db-4aaf-b996-c34cbeff8227] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:18:28.163 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5896807875828437) > weight (0.25) 2024-09-14T23:18:28.163 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5862390550376555) > weight (0.25) 2024-09-14T23:18:28.163 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9401019023136747) > weight (0.25) 2024-09-14T23:18:28.163 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-14T23:18:28.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-14T23:18:28.164 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-14T23:18:28.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:18:28.163+0000 7fddee05b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 54029) UID: 1000 2024-09-14T23:18:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T23:18:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-14T23:18:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ tee -a /tmp/tmp.OxLGv2e3xU/rbd-mirror.test.log 2024-09-14T23:18:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ date '+%F %T' 2024-09-14T23:18:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo 2024-09-14 23:18:28 TEST: image deletions should propagate 2024-09-14T23:18:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:2024-09-14 23:18:28 TEST: image deletions should propagate 2024-09-14T23:18:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-14T23:18:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_count=0 2024-09-14T23:18:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local count 2024-09-14T23:18:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local last_count=0 2024-09-14T23:18:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ true 2024-09-14T23:18:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 40 2024-09-14T23:18:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in `seq 1 40` 2024-09-14T23:18:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 1 -ne 1 2024-09-14T23:18:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T23:18:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep 'images: ' 2024-09-14T23:18:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cut '-d ' -f 2 2024-09-14T23:18:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T23:18:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ count=0 2024-09-14T23:18:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 = 0 2024-09-14T23:18:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-14T23:18:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state 2024-09-14T23:18:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 40 2024-09-14T23:18:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in `seq 1 40` 2024-09-14T23:18:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 1 -ne 1 2024-09-14T23:18:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T23:18:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep 'image health:' 2024-09-14T23:18:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cut '-d ' -f 3 2024-09-14T23:18:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T23:18:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ state=OK 2024-09-14T23:18:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test OK = ERROR 2024-09-14T23:18:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test OK = OK 2024-09-14T23:18:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 50 2024-09-14T23:18:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_1 2024-09-14T23:18:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-14T23:18:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T23:18:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_1 2024-09-14T23:18:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T23:18:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T23:18:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-14T23:18:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T23:18:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_1 2024-09-14T23:18:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T23:18:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T23:18:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_2 2024-09-14T23:18:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-14T23:18:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_2 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T23:18:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T23:18:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_2 2024-09-14T23:18:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T23:18:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T23:18:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_3 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_3 2024-09-14T23:18:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T23:18:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T23:18:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T23:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_3 2024-09-14T23:18:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T23:18:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T23:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_4 2024-09-14T23:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-14T23:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_4 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T23:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T23:18:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-14T23:18:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:18:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T23:18:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_4 2024-09-14T23:18:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T23:18:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T23:18:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_5 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_5 2024-09-14T23:18:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T23:18:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.546 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T23:18:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_5 2024-09-14T23:18:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T23:18:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_6 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_6 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T23:18:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T23:18:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-14T23:18:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:18:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T23:18:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_6 2024-09-14T23:18:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T23:18:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T23:18:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_7 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T23:18:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_7 2024-09-14T23:18:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T23:18:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T23:18:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-14T23:18:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:18:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T23:18:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_7 2024-09-14T23:18:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T23:18:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_8 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T23:18:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_8 2024-09-14T23:18:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T23:18:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T23:18:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-14T23:18:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_8 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T23:18:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T23:18:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_9 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T23:18:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_9 2024-09-14T23:18:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T23:18:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T23:18:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-14T23:18:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_9 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T23:18:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T23:18:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_10 2024-09-14T23:18:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-14T23:18:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_10 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T23:18:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T23:18:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_10 2024-09-14T23:18:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T23:18:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T23:18:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_11 2024-09-14T23:18:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-14T23:18:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:18:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_11 2024-09-14T23:18:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_11 2024-09-14T23:18:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-14T23:18:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-14T23:18:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_11 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_11 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_11 2024-09-14T23:18:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-14T23:18:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-14T23:18:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_12 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_12 2024-09-14T23:18:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_12 2024-09-14T23:18:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-14T23:18:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-14T23:18:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-14T23:18:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_12 2024-09-14T23:18:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_12 2024-09-14T23:18:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_12 2024-09-14T23:18:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-14T23:18:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-14T23:18:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_13 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_13 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_13 2024-09-14T23:18:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-14T23:18:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-14T23:18:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_13 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_13 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_13 2024-09-14T23:18:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-14T23:18:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-14T23:18:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_14 2024-09-14T23:18:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-14T23:18:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_14 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_14 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-14T23:18:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-14T23:18:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-14T23:18:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_14 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_14 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_14 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-14T23:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_15 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_15 2024-09-14T23:18:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_15 2024-09-14T23:18:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-14T23:18:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-14T23:18:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-14T23:18:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_15 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_15 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_15 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-14T23:18:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-14T23:18:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_16 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_16 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_16 2024-09-14T23:18:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-14T23:18:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-14T23:18:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-14T23:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_16 2024-09-14T23:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_16 2024-09-14T23:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_16 2024-09-14T23:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-14T23:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_17 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:18:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_17 2024-09-14T23:18:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_17 2024-09-14T23:18:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-14T23:18:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-14T23:18:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-14T23:18:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_17 2024-09-14T23:18:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_17 2024-09-14T23:18:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_17 2024-09-14T23:18:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-14T23:18:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-14T23:18:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_18 2024-09-14T23:18:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-14T23:18:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_18 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_18 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-14T23:18:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-14T23:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-14T23:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_18 2024-09-14T23:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_18 2024-09-14T23:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_18 2024-09-14T23:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-14T23:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-14T23:18:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_19 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_19 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_19 2024-09-14T23:18:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-14T23:18:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-14T23:18:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_19 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_19 2024-09-14T23:18:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_19 2024-09-14T23:18:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-14T23:18:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-14T23:18:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_20 2024-09-14T23:18:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-14T23:18:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_20 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_20 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-14T23:18:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-14T23:18:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_20 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_20 2024-09-14T23:18:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_20 2024-09-14T23:18:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-14T23:18:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-14T23:18:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_21 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_21 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_21 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-14T23:18:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-14T23:18:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-14T23:18:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_21 2024-09-14T23:18:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_21 2024-09-14T23:18:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_21 2024-09-14T23:18:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-14T23:18:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-14T23:18:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_22 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_22 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_22 2024-09-14T23:18:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-14T23:18:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-14T23:18:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-14T23:18:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_22 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_22 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_22 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-14T23:18:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-14T23:18:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_23 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_23 2024-09-14T23:18:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_23 2024-09-14T23:18:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-14T23:18:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-14T23:18:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-14T23:18:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_23 2024-09-14T23:18:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_23 2024-09-14T23:18:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_23 2024-09-14T23:18:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-14T23:18:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-14T23:18:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_24 2024-09-14T23:18:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_24 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_24 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-14T23:18:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-14T23:18:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-14T23:18:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_24 2024-09-14T23:18:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_24 2024-09-14T23:18:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_24 2024-09-14T23:18:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-14T23:18:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_25 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:18:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_25 2024-09-14T23:18:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_25 2024-09-14T23:18:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-14T23:18:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-14T23:18:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-14T23:18:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_25 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_25 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_25 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-14T23:18:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-14T23:18:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_26 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_26 2024-09-14T23:18:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_26 2024-09-14T23:18:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-14T23:18:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-14T23:18:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-14T23:18:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_26 2024-09-14T23:18:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_26 2024-09-14T23:18:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_26 2024-09-14T23:18:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-14T23:18:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_27 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_27 2024-09-14T23:18:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_27 2024-09-14T23:18:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-14T23:18:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-14T23:18:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_27 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_27 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_27 2024-09-14T23:18:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-14T23:18:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-14T23:18:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_28 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:18:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_28 2024-09-14T23:18:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_28 2024-09-14T23:18:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-14T23:18:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-14T23:18:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-14T23:18:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_28 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_28 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_28 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-14T23:18:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_29 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_29 2024-09-14T23:18:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_29 2024-09-14T23:18:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-14T23:18:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-14T23:18:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-14T23:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_29 2024-09-14T23:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_29 2024-09-14T23:18:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_29 2024-09-14T23:18:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-14T23:18:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-14T23:18:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_30 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_30 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_30 2024-09-14T23:18:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-14T23:18:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-14T23:18:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-14T23:18:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_30 2024-09-14T23:18:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_30 2024-09-14T23:18:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_30 2024-09-14T23:18:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-14T23:18:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_31 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_31 2024-09-14T23:18:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_31 2024-09-14T23:18:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-14T23:18:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-14T23:18:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-14T23:18:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_31 2024-09-14T23:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_31 2024-09-14T23:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_31 2024-09-14T23:18:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-14T23:18:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-14T23:18:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_32 2024-09-14T23:18:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-14T23:18:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:18:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_32 2024-09-14T23:18:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_32 2024-09-14T23:18:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-14T23:18:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-14T23:18:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-14T23:18:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_32 2024-09-14T23:18:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_32 2024-09-14T23:18:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_32 2024-09-14T23:18:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-14T23:18:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-14T23:18:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_33 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_33 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_33 2024-09-14T23:18:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-14T23:18:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-14T23:18:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_33 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_33 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_33 2024-09-14T23:18:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-14T23:18:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-14T23:18:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_34 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_34 2024-09-14T23:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_34 2024-09-14T23:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-14T23:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-14T23:18:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_34 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_34 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_34 2024-09-14T23:18:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-14T23:18:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-14T23:18:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_35 2024-09-14T23:18:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_35 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_35 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-14T23:18:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-14T23:18:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_35 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_35 2024-09-14T23:18:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_35 2024-09-14T23:18:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-14T23:18:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-14T23:18:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_36 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_36 2024-09-14T23:18:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_36 2024-09-14T23:18:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-14T23:18:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-14T23:18:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-14T23:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_36 2024-09-14T23:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_36 2024-09-14T23:18:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_36 2024-09-14T23:18:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-14T23:18:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-14T23:18:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_37 2024-09-14T23:18:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-14T23:18:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_37 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_37 2024-09-14T23:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-14T23:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-14T23:18:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-14T23:18:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_37 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_37 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_37 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-14T23:18:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-14T23:18:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_38 2024-09-14T23:18:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-14T23:18:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_38 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_38 2024-09-14T23:18:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-14T23:18:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-14T23:18:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_38 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_38 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_38 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-14T23:18:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-14T23:18:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_39 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_39 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_39 2024-09-14T23:18:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-14T23:18:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-14T23:18:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-14T23:18:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_39 2024-09-14T23:18:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_39 2024-09-14T23:18:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_39 2024-09-14T23:18:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-14T23:18:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-14T23:18:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_40 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_40 2024-09-14T23:18:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_40 2024-09-14T23:18:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-14T23:18:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-14T23:18:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:36.165 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-14T23:18:36.165 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:18:36.165 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-14T23:18:36.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-14T23:18:36.165 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-14T23:18:36.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-14T23:18:36.169 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-09-14T23:18:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_40 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_40 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_40 2024-09-14T23:18:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-14T23:18:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_41 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_41 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_41 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-14T23:18:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-14T23:18:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_41 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_41 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_41 2024-09-14T23:18:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-14T23:18:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-14T23:18:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_42 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_42 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_42 2024-09-14T23:18:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-14T23:18:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-14T23:18:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_42 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_42 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_42 2024-09-14T23:18:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-14T23:18:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-14T23:18:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_43 2024-09-14T23:18:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-14T23:18:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_43 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_43 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-14T23:18:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-14T23:18:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_43 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_43 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_43 2024-09-14T23:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-14T23:18:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_44 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_44 2024-09-14T23:18:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_44 2024-09-14T23:18:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-14T23:18:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-14T23:18:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_44 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_44 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_44 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-14T23:18:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-14T23:18:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_45 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_45 2024-09-14T23:18:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_45 2024-09-14T23:18:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-14T23:18:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-14T23:18:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_45 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_45 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_45 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-14T23:18:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_46 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_46 2024-09-14T23:18:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_46 2024-09-14T23:18:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-14T23:18:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-14T23:18:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-14T23:18:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_46 2024-09-14T23:18:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_46 2024-09-14T23:18:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_46 2024-09-14T23:18:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-14T23:18:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-14T23:18:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_47 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_47 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_47 2024-09-14T23:18:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-14T23:18:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-14T23:18:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-14T23:18:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_47 2024-09-14T23:18:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_47 2024-09-14T23:18:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_47 2024-09-14T23:18:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-14T23:18:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-14T23:18:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_48 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_48 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_48 2024-09-14T23:18:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-14T23:18:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-14T23:18:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-14T23:18:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_48 2024-09-14T23:18:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_48 2024-09-14T23:18:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_48 2024-09-14T23:18:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-14T23:18:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-14T23:18:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_49 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:18:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_49 2024-09-14T23:18:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_49 2024-09-14T23:18:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-14T23:18:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-14T23:18:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-14T23:18:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_49 2024-09-14T23:18:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_49 2024-09-14T23:18:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_49 2024-09-14T23:18:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-14T23:18:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-14T23:18:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_50 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_50 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_50 2024-09-14T23:18:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-14T23:18:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-14T23:18:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-14T23:18:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_50 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_50 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_50 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-14T23:18:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-14T23:18:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-14T23:18:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ tee -a /tmp/tmp.OxLGv2e3xU/rbd-mirror.test.log 2024-09-14T23:18:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ date '+%F %T' 2024-09-14T23:18:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo 2024-09-14 23:18:37 TEST: delete images during bootstrap 2024-09-14T23:18:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:2024-09-14 23:18:37 TEST: delete images during bootstrap 2024-09-14T23:18:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-14T23:18:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=image 2024-09-14T23:18:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-14T23:18:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd: mirroring is already configured for image mode 2024-09-14T23:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-14T23:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=image 2024-09-14T23:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-14T23:18:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd: mirroring is already configured for image mode 2024-09-14T23:18:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ start_mirror cluster1 2024-09-14T23:18:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local val=cluster1 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster_var_name=cluster 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local instance_var_name=instance 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cluster=cluster1 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=cluster1 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ instance=0 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval cluster=cluster1 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ cluster=cluster1 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ eval instance=0 2024-09-14T23:18:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ instance=0 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=test 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 1 10 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_1 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:18:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:18:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:18:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:18:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:18:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_1 2024-09-14T23:18:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-14T23:18:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:18:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-14T23:18:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:18:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-14T23:18:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_1': 2024-09-14T23:18:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:18:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:18:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 18ad424182b1 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.18ad424182b1 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:18:38 2024 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:18:38 2024 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:18:38 2024 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 1079c482-754b-40dd-9529-1035301119b5 2024-09-14T23:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:18:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-14T23:18:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:18:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=7 2024-09-14T23:18:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:18:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:18:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 26320896 bytes 2024-09-14T23:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 62278656 bytes 2024-09-14T23:18:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 84516864 bytes 2024-09-14T23:18:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 174295040 bytes 2024-09-14T23:18:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 197797888 bytes 2024-09-14T23:18:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 227093504 bytes 2024-09-14T23:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 268462080 bytes 2024-09-14T23:18:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 286306304 bytes 2024-09-14T23:18:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 372968448 bytes 2024-09-14T23:18:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 401699840 bytes 2024-09-14T23:18:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 469161984 bytes 2024-09-14T23:18:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 573982720 bytes 2024-09-14T23:18:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 658384896 bytes 2024-09-14T23:18:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 697167872 bytes 2024-09-14T23:18:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 754711552 bytes 2024-09-14T23:18:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 808812544 bytes 2024-09-14T23:18:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 881939456 bytes 2024-09-14T23:18:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 938725376 bytes 2024-09-14T23:18:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 959662080 bytes 2024-09-14T23:18:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 1015552000 bytes 2024-09-14T23:18:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 1065775104 bytes 2024-09-14T23:18:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1080263680 bytes 2024-09-14T23:18:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1157347328 bytes 2024-09-14T23:18:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1221083136 bytes 2024-09-14T23:18:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1302099968 bytes 2024-09-14T23:18:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1328735232 bytes 2024-09-14T23:18:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1367485440 bytes 2024-09-14T23:18:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1416117248 bytes 2024-09-14T23:18:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1466513408 bytes 2024-09-14T23:18:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1498139648 bytes 2024-09-14T23:18:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1544249344 bytes 2024-09-14T23:18:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1549239296 bytes 2024-09-14T23:18:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1550125056 bytes 2024-09-14T23:18:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1550742528 bytes 2024-09-14T23:18:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1587271680 bytes 2024-09-14T23:18:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1626258432 bytes 2024-09-14T23:18:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1628603392 bytes 2024-09-14T23:18:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1673427968 bytes 2024-09-14T23:18:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1695098880 bytes 2024-09-14T23:18:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1746301952 bytes 2024-09-14T23:18:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1770888192 bytes 2024-09-14T23:18:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1793293312 bytes 2024-09-14T23:18:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1816585216 bytes 2024-09-14T23:18:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1875313664 bytes 2024-09-14T23:18:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1932553216 bytes 2024-09-14T23:18:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2001441792 bytes 2024-09-14T23:18:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2022082560 bytes 2024-09-14T23:18:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2083517440 bytes 2024-09-14T23:18:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2095538176 bytes 2024-09-14T23:18:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2131368960 bytes 2024-09-14T23:18:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2180999168 bytes 2024-09-14T23:18:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2197673984 bytes 2024-09-14T23:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2236825600 bytes 2024-09-14T23:18:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2253118464 bytes 2024-09-14T23:18:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2365663232 bytes 2024-09-14T23:18:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2388403200 bytes 2024-09-14T23:18:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2452537344 bytes 2024-09-14T23:18:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2462584832 bytes 2024-09-14T23:18:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2517029888 bytes 2024-09-14T23:18:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:18:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T23:18:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_1 2024-09-14T23:18:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T23:18:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id=12a07834c6db 2024-09-14T23:18:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_1 present 12a07834c6db 2024-09-14T23:18:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id=12a07834c6db 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_1 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T23:18:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=12a07834c6db 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 12a07834c6db 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z 12a07834c6db -o 12a07834c6db = 12a07834c6db 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-14T23:18:46.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:18:46.640+0000 7ff808ade640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5573df5da800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2024-09-14T23:18:46.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:18:46.640+0000 7ff808ade640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5573df5da800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2024-09-14T23:18:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:18:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-14T23:18:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:18:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:18:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T23:18:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_1 2024-09-14T23:18:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T23:18:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T23:18:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:18:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:18:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:18:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-14T23:18:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:18:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:18:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:18:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:18:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:18:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T23:18:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:18:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:18:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_1 2024-09-14T23:18:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T23:18:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:18:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T23:18:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:18:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:18:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:18:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:18:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-14T23:18:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-14T23:18:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-14T23:18:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:18:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T23:18:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T23:18:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T23:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:18:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:18:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T23:18:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T23:18:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T23:18:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:18:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:18:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:18:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:18:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T23:18:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T23:18:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T23:18:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:18:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:18:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:18:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:18:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T23:18:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:18:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:18:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:18:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T23:18:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T23:18:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:18:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:18:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:19:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T23:19:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:19:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T23:19:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T23:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:19:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:19:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:19:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T23:19:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_1 2024-09-14T23:19:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T23:19:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T23:19:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:19:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:19:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:19:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_2 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:19:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_2 2024-09-14T23:19:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-14T23:19:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:19:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-14T23:19:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:19:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-14T23:19:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_2': 2024-09-14T23:19:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:19:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:19:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:19:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 18e3c46c3314 2024-09-14T23:19:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.18e3c46c3314 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:19:23 2024 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:19:23 2024 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:19:23 2024 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 28af123a-05d9-4c59-8884-e04418428c4b 2024-09-14T23:19:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:19:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-14T23:19:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:19:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=14 2024-09-14T23:19:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:19:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:19:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 15335424 bytes 2024-09-14T23:19:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 37915648 bytes 2024-09-14T23:19:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 50595840 bytes 2024-09-14T23:19:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 94628864 bytes 2024-09-14T23:19:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 104043520 bytes 2024-09-14T23:19:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 134057984 bytes 2024-09-14T23:19:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 216337408 bytes 2024-09-14T23:19:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 248038400 bytes 2024-09-14T23:19:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 273046528 bytes 2024-09-14T23:19:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 350194688 bytes 2024-09-14T23:19:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 381302784 bytes 2024-09-14T23:19:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 416822272 bytes 2024-09-14T23:19:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 496776192 bytes 2024-09-14T23:19:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 533174272 bytes 2024-09-14T23:19:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 559447040 bytes 2024-09-14T23:19:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 560077824 bytes 2024-09-14T23:19:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 564022272 bytes 2024-09-14T23:19:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 637469696 bytes 2024-09-14T23:19:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 676788224 bytes 2024-09-14T23:19:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 704627712 bytes 2024-09-14T23:19:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 738381824 bytes 2024-09-14T23:19:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 789314560 bytes 2024-09-14T23:19:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 789926912 bytes 2024-09-14T23:19:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 836389888 bytes 2024-09-14T23:19:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 896763904 bytes 2024-09-14T23:19:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 919374848 bytes 2024-09-14T23:19:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 958084096 bytes 2024-09-14T23:19:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 979729408 bytes 2024-09-14T23:19:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1028740096 bytes 2024-09-14T23:19:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1063094272 bytes 2024-09-14T23:19:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1091013632 bytes 2024-09-14T23:19:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1144538112 bytes 2024-09-14T23:19:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1163505664 bytes 2024-09-14T23:19:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1211159552 bytes 2024-09-14T23:19:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1211751424 bytes 2024-09-14T23:19:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1230529536 bytes 2024-09-14T23:19:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1300637696 bytes 2024-09-14T23:19:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1381226496 bytes 2024-09-14T23:19:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1459794944 bytes 2024-09-14T23:19:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1460408320 bytes 2024-09-14T23:19:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1488077824 bytes 2024-09-14T23:19:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1552186368 bytes 2024-09-14T23:19:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1677779968 bytes 2024-09-14T23:19:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1701078016 bytes 2024-09-14T23:19:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1704602624 bytes 2024-09-14T23:19:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1756243968 bytes 2024-09-14T23:19:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1796037632 bytes 2024-09-14T23:19:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1857105920 bytes 2024-09-14T23:19:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1886788608 bytes 2024-09-14T23:19:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1923095552 bytes 2024-09-14T23:19:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 1942325248 bytes 2024-09-14T23:19:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2003244032 bytes 2024-09-14T23:19:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2029760512 bytes 2024-09-14T23:19:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2063775744 bytes 2024-09-14T23:19:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2085934080 bytes 2024-09-14T23:19:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2086536192 bytes 2024-09-14T23:19:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2128497664 bytes 2024-09-14T23:19:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2174988288 bytes 2024-09-14T23:19:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2262194176 bytes 2024-09-14T23:19:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2415422464 bytes 2024-09-14T23:19:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2422435840 bytes 2024-09-14T23:19:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2440547328 bytes 2024-09-14T23:19:32.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2477651968 bytes 2024-09-14T23:19:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2533061632 bytes 2024-09-14T23:19:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2619058176 bytes 2024-09-14T23:19:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2629398528 bytes 2024-09-14T23:19:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2731823104 bytes 2024-09-14T23:19:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2783862784 bytes 2024-09-14T23:19:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2895361024 bytes 2024-09-14T23:19:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3024470016 bytes 2024-09-14T23:19:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3063403520 bytes 2024-09-14T23:19:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3086839808 bytes 2024-09-14T23:19:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3159306240 bytes 2024-09-14T23:19:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3206042624 bytes 2024-09-14T23:19:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3211704320 bytes 2024-09-14T23:19:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3239683072 bytes 2024-09-14T23:19:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3266430976 bytes 2024-09-14T23:19:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3311062016 bytes 2024-09-14T23:19:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3379829760 bytes 2024-09-14T23:19:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3516758016 bytes 2024-09-14T23:19:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3539753984 bytes 2024-09-14T23:19:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3625084928 bytes 2024-09-14T23:19:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3643986944 bytes 2024-09-14T23:19:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3685126144 bytes 2024-09-14T23:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3722283008 bytes 2024-09-14T23:19:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3729176576 bytes 2024-09-14T23:19:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3780521984 bytes 2024-09-14T23:19:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3805377536 bytes 2024-09-14T23:19:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3846106112 bytes 2024-09-14T23:19:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3846695936 bytes 2024-09-14T23:19:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3891382272 bytes 2024-09-14T23:19:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3895869440 bytes 2024-09-14T23:19:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3995708416 bytes 2024-09-14T23:19:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4026858496 bytes 2024-09-14T23:19:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4086636544 bytes 2024-09-14T23:19:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4114553856 bytes 2024-09-14T23:19:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4120835072 bytes 2024-09-14T23:19:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4123995136 bytes 2024-09-14T23:19:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4140736512 bytes 2024-09-14T23:19:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4170336256 bytes 2024-09-14T23:19:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4176560128 bytes 2024-09-14T23:19:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4248607744 bytes 2024-09-14T23:19:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4307258368 bytes 2024-09-14T23:19:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4323341312 bytes 2024-09-14T23:19:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4359118848 bytes 2024-09-14T23:19:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4425931776 bytes 2024-09-14T23:19:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4438126592 bytes 2024-09-14T23:19:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4487220224 bytes 2024-09-14T23:19:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 4513586176 bytes 2024-09-14T23:19:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11000 IOs, 4514200576 bytes 2024-09-14T23:19:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11100 IOs, 4579592192 bytes 2024-09-14T23:19:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11200 IOs, 4621256704 bytes 2024-09-14T23:19:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11300 IOs, 4676506624 bytes 2024-09-14T23:19:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11400 IOs, 4740068352 bytes 2024-09-14T23:19:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11500 IOs, 4809306112 bytes 2024-09-14T23:19:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11600 IOs, 4851724288 bytes 2024-09-14T23:19:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11700 IOs, 4852335616 bytes 2024-09-14T23:19:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:19:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:19:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:19:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:19:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:19:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_2 2024-09-14T23:19:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T23:19:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:19:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id=12a0ded3598 2024-09-14T23:19:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:19:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:19:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_2 present 12a0ded3598 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id=12a0ded3598 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:19:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_2 2024-09-14T23:19:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T23:19:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=12a0ded3598 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 12a0ded3598 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z 12a0ded3598 -o 12a0ded3598 = 12a0ded3598 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-14T23:19:39.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:19:39.697+0000 7ff808ade640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5573dc7a1800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-09-14T23:19:39.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:19:39.697+0000 7ff808ade640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5573dc7a1800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=188, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-09-14T23:19:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-14T23:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T23:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_2 2024-09-14T23:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T23:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:19:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T23:19:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:19:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:19:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:19:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-14T23:19:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:19:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_2 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T23:19:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:19:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T23:19:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:19:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:19:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:19:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:19:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-14T23:19:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-14T23:19:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-14T23:19:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:19:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:19:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T23:19:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T23:19:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T23:19:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:19:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:19:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:19:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:19:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:19:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T23:19:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T23:19:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T23:19:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:19:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:19:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:19:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:19:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:19:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T23:19:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T23:19:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T23:19:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:19:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:19:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:19:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:19:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:19:52.163 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9340973475942068) > weight (0.25) 2024-09-14T23:19:52.164 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.31206524026997584) > weight (0.25) 2024-09-14T23:19:52.164 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-14T23:19:52.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-14T23:19:52.164 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-14T23:19:52.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:19:52.164+0000 7f9c410f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 62006) UID: 1000 2024-09-14T23:19:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T23:19:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:19:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:19:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:19:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T23:19:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T23:19:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:19:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:19:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:19:52.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:19:52.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:19:59.165 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-14T23:19:59.165 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:19:59.166 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-14T23:19:59.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-14T23:19:59.166 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-14T23:19:59.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-14T23:19:59.169 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-09-14T23:20:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T23:20:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:20:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T23:20:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T23:20:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:20:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:20:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:20:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T23:20:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_2 2024-09-14T23:20:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T23:20:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T23:20:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:20:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:20:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:20:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_3 2024-09-14T23:20:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-14T23:20:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:20:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:20:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:20:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:20:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:20:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:20:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:20:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_3 2024-09-14T23:20:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-14T23:20:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:20:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-14T23:20:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:20:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-14T23:20:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_3': 2024-09-14T23:20:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:20:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:20:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:20:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 191f3185ab59 2024-09-14T23:20:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.191f3185ab59 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:20:16 2024 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:20:16 2024 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:20:16 2024 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 940b30c6-8fa3-4898-83ec-ba36f31499ac 2024-09-14T23:20:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:20:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-14T23:20:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:20:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=8 2024-09-14T23:20:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:20:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:20:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 77452288 bytes 2024-09-14T23:20:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 116079616 bytes 2024-09-14T23:20:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 170616832 bytes 2024-09-14T23:20:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 221327360 bytes 2024-09-14T23:20:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 223877120 bytes 2024-09-14T23:20:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 268351488 bytes 2024-09-14T23:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 303126528 bytes 2024-09-14T23:20:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 341052416 bytes 2024-09-14T23:20:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 409821184 bytes 2024-09-14T23:20:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 457351168 bytes 2024-09-14T23:20:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 482409472 bytes 2024-09-14T23:20:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 506249216 bytes 2024-09-14T23:20:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 567717888 bytes 2024-09-14T23:20:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 610322432 bytes 2024-09-14T23:20:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 690678784 bytes 2024-09-14T23:20:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 733352960 bytes 2024-09-14T23:20:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 757880832 bytes 2024-09-14T23:20:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 792170496 bytes 2024-09-14T23:20:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 828883968 bytes 2024-09-14T23:20:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 881619968 bytes 2024-09-14T23:20:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 961268736 bytes 2024-09-14T23:20:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1010111488 bytes 2024-09-14T23:20:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1031956480 bytes 2024-09-14T23:20:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1074448384 bytes 2024-09-14T23:20:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1122693120 bytes 2024-09-14T23:20:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1191144448 bytes 2024-09-14T23:20:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1210535936 bytes 2024-09-14T23:20:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1327344640 bytes 2024-09-14T23:20:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1395209216 bytes 2024-09-14T23:20:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1428104192 bytes 2024-09-14T23:20:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1431156736 bytes 2024-09-14T23:20:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1471124480 bytes 2024-09-14T23:20:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1518819328 bytes 2024-09-14T23:20:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1537667072 bytes 2024-09-14T23:20:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1575507968 bytes 2024-09-14T23:20:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1609041920 bytes 2024-09-14T23:20:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1655776256 bytes 2024-09-14T23:20:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1707180032 bytes 2024-09-14T23:20:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1746272256 bytes 2024-09-14T23:20:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1791782912 bytes 2024-09-14T23:20:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1792408576 bytes 2024-09-14T23:20:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1869319168 bytes 2024-09-14T23:20:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1875421184 bytes 2024-09-14T23:20:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1968018432 bytes 2024-09-14T23:20:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2046914560 bytes 2024-09-14T23:20:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2103024640 bytes 2024-09-14T23:20:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2189691904 bytes 2024-09-14T23:20:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2247584768 bytes 2024-09-14T23:20:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2260643840 bytes 2024-09-14T23:20:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2322619392 bytes 2024-09-14T23:20:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2341782528 bytes 2024-09-14T23:20:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2408628224 bytes 2024-09-14T23:20:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2467938304 bytes 2024-09-14T23:20:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2554810368 bytes 2024-09-14T23:20:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2595206144 bytes 2024-09-14T23:20:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2636516352 bytes 2024-09-14T23:20:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2677879808 bytes 2024-09-14T23:20:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2707324928 bytes 2024-09-14T23:20:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2735647744 bytes 2024-09-14T23:20:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2744530944 bytes 2024-09-14T23:20:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2765546496 bytes 2024-09-14T23:20:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2834556928 bytes 2024-09-14T23:20:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2921303040 bytes 2024-09-14T23:20:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2925928448 bytes 2024-09-14T23:20:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:20:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:20:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:20:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_3 2024-09-14T23:20:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T23:20:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:20:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id=12a0e1925c9a 2024-09-14T23:20:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:20:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:20:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_3 present 12a0e1925c9a 2024-09-14T23:20:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id=12a0e1925c9a 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_3 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T23:20:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:20:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=12a0e1925c9a 2024-09-14T23:20:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 12a0e1925c9a 2024-09-14T23:20:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z 12a0e1925c9a -o 12a0e1925c9a = 12a0e1925c9a 2024-09-14T23:20:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:20:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:20:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:20:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-14T23:20:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-14T23:20:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:20:26.441+0000 7ff808ade640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5573db87e400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-09-14T23:20:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:20:26.441+0000 7ff808ade640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5573db87e400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=189, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-09-14T23:20:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:20:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-14T23:20:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:20:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:20:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:20:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:20:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:20:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T23:20:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:20:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:20:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_3 2024-09-14T23:20:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T23:20:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:20:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T23:20:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:20:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:20:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:20:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-14T23:20:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:20:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_3 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T23:20:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:20:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T23:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-14T23:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-14T23:20:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-14T23:20:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:20:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:20:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T23:20:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T23:20:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T23:20:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:20:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:20:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:20:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:20:31.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:20:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T23:20:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T23:20:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T23:20:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:20:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:20:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:20:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:20:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:20:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T23:20:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T23:20:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T23:20:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:20:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:20:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:20:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:20:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:20:35.169 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6148293086459109) > weight (0.25) 2024-09-14T23:20:35.170 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-14T23:20:35.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-14T23:20:35.170 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-14T23:20:35.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:20:35.170+0000 7f1fb7345640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 60775) UID: 1000 2024-09-14T23:20:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T23:20:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T23:20:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T23:20:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:20:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:20:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:20:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:20:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:20:44.171 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-14T23:20:44.171 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:20:44.172 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-14T23:20:44.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T23:20:44.172 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T23:20:44.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T23:20:44.175 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-14T23:20:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T23:20:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:20:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:20:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:20:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T23:20:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T23:20:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:20:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:20:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:20:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:20:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:21:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T23:21:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_3 2024-09-14T23:21:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T23:21:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T23:21:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_4 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:21:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:21:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:21:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:21:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:21:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_4 2024-09-14T23:21:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-14T23:21:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:21:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-14T23:21:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:21:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-14T23:21:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_4': 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 195edbd6d053 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.195edbd6d053 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:21:03 2024 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:21:03 2024 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:21:03 2024 2024-09-14T23:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:21:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:21:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: a628ec7a-ca97-45b4-aedd-c79c72b32bce 2024-09-14T23:21:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:21:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-14T23:21:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:21:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=8 2024-09-14T23:21:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:21:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:21:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 64232448 bytes 2024-09-14T23:21:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 103414784 bytes 2024-09-14T23:21:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 162412544 bytes 2024-09-14T23:21:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 224956416 bytes 2024-09-14T23:21:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 273589248 bytes 2024-09-14T23:21:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 329461760 bytes 2024-09-14T23:21:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 335989760 bytes 2024-09-14T23:21:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 424481792 bytes 2024-09-14T23:21:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 454769664 bytes 2024-09-14T23:21:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 464360448 bytes 2024-09-14T23:21:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 508616704 bytes 2024-09-14T23:21:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 590388224 bytes 2024-09-14T23:21:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 634443776 bytes 2024-09-14T23:21:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 658638848 bytes 2024-09-14T23:21:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 693244928 bytes 2024-09-14T23:21:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 753627136 bytes 2024-09-14T23:21:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 793819136 bytes 2024-09-14T23:21:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 828910592 bytes 2024-09-14T23:21:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 864504832 bytes 2024-09-14T23:21:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 865097728 bytes 2024-09-14T23:21:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 941338624 bytes 2024-09-14T23:21:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 941949952 bytes 2024-09-14T23:21:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 998850560 bytes 2024-09-14T23:21:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1030156288 bytes 2024-09-14T23:21:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1131473920 bytes 2024-09-14T23:21:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1167850496 bytes 2024-09-14T23:21:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1199970304 bytes 2024-09-14T23:21:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1257995264 bytes 2024-09-14T23:21:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1278764032 bytes 2024-09-14T23:21:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1299945472 bytes 2024-09-14T23:21:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1341797376 bytes 2024-09-14T23:21:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1358300160 bytes 2024-09-14T23:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1456477184 bytes 2024-09-14T23:21:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1497020416 bytes 2024-09-14T23:21:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1537643520 bytes 2024-09-14T23:21:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1611952128 bytes 2024-09-14T23:21:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1690509312 bytes 2024-09-14T23:21:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1707799552 bytes 2024-09-14T23:21:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1817645056 bytes 2024-09-14T23:21:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1866970112 bytes 2024-09-14T23:21:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1922912256 bytes 2024-09-14T23:21:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1939422208 bytes 2024-09-14T23:21:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1974231040 bytes 2024-09-14T23:21:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 2073404416 bytes 2024-09-14T23:21:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2085379072 bytes 2024-09-14T23:21:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2085992448 bytes 2024-09-14T23:21:10.335 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2137975808 bytes 2024-09-14T23:21:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2163362816 bytes 2024-09-14T23:21:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2182643712 bytes 2024-09-14T23:21:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2232440832 bytes 2024-09-14T23:21:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2275258368 bytes 2024-09-14T23:21:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2303866880 bytes 2024-09-14T23:21:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2338692096 bytes 2024-09-14T23:21:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2417776640 bytes 2024-09-14T23:21:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2469633024 bytes 2024-09-14T23:21:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2522237952 bytes 2024-09-14T23:21:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2551879680 bytes 2024-09-14T23:21:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2588360704 bytes 2024-09-14T23:21:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2703802368 bytes 2024-09-14T23:21:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2754436096 bytes 2024-09-14T23:21:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2778291200 bytes 2024-09-14T23:21:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2795054080 bytes 2024-09-14T23:21:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2867964928 bytes 2024-09-14T23:21:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2868581376 bytes 2024-09-14T23:21:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2902084608 bytes 2024-09-14T23:21:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2943510528 bytes 2024-09-14T23:21:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2954008576 bytes 2024-09-14T23:21:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:21:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:21:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:21:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:21:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_4 2024-09-14T23:21:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T23:21:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:21:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id=12a0fb6da19e 2024-09-14T23:21:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:21:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_4 present 12a0fb6da19e 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id=12a0fb6da19e 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_4 2024-09-14T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:21:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=12a0fb6da19e 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 12a0fb6da19e 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z 12a0fb6da19e -o 12a0fb6da19e = 12a0fb6da19e 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-14T23:21:13.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:21:13.658+0000 7ff808ade640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5573df652400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=188 2024-09-14T23:21:13.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:21:13.658+0000 7ff808ade640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5573df652400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=190, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=bc, last_copied_object_number=128, snap_seqs={188=18446744073709551614}] 2024-09-14T23:21:15.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:21:15.158+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfbedf380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:21:15.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:21:15.158+0000 7fb3e21e7640 -1 rbd::mirror::ImageReplayer: 0x558cfc414c80 [3/a628ec7a-ca97-45b4-aedd-c79c72b32bce] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:21:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:21:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:21:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_4 2024-09-14T23:21:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T23:21:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:21:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T23:21:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:21:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:21:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:21:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-14T23:21:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:21:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:21:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:21:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:21:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:21:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T23:21:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:21:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:21:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_4 2024-09-14T23:21:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T23:21:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:21:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T23:21:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:21:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:21:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:21:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:21:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-14T23:21:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-14T23:21:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-14T23:21:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:21:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:21:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T23:21:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T23:21:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T23:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:21:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T23:21:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T23:21:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T23:21:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:21:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:21:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:21:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:21:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:21:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T23:21:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T23:21:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T23:21:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:21:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:21:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:21:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:21:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:21:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T23:21:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T23:21:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T23:21:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:21:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:21:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:21:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:21:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:21:34.170 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9311633131311664) > weight (0.25) 2024-09-14T23:21:34.171 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.39952426813075825) > weight (0.25) 2024-09-14T23:21:34.171 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8722390761432036) > weight (0.25) 2024-09-14T23:21:34.171 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-14T23:21:34.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-14T23:21:34.171 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T23:21:34.172 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-14T23:21:34.172 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:21:34.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:21:34.172+0000 7ff8162f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 48676) UID: 1000 2024-09-14T23:21:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T23:21:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T23:21:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T23:21:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:21:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:21:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:21:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:21:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:21:40.274 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-14T23:21:40.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T23:21:40.274 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T23:21:40.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T23:21:40.278 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-09-14T23:21:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T23:21:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_4 2024-09-14T23:21:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T23:21:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T23:21:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_5 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:21:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:21:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:21:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:21:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_5 2024-09-14T23:21:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-14T23:21:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:21:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-14T23:21:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:21:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_5': 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 19998230d4b0 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.19998230d4b0 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:21:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:21:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:21:50 2024 2024-09-14T23:21:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:21:50 2024 2024-09-14T23:21:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:21:50 2024 2024-09-14T23:21:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:21:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:21:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 01b74956-f3e1-412e-bf2e-a04d6c89c901 2024-09-14T23:21:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:21:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-14T23:21:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:21:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:21:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:21:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:21:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=11 2024-09-14T23:21:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:21:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:21:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 62687232 bytes 2024-09-14T23:21:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 104461312 bytes 2024-09-14T23:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 194671616 bytes 2024-09-14T23:21:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 216916992 bytes 2024-09-14T23:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 220411904 bytes 2024-09-14T23:21:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 231217152 bytes 2024-09-14T23:21:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 264401920 bytes 2024-09-14T23:21:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 288079872 bytes 2024-09-14T23:21:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 288944128 bytes 2024-09-14T23:21:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 317243392 bytes 2024-09-14T23:21:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 392460288 bytes 2024-09-14T23:21:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 406627328 bytes 2024-09-14T23:21:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 467381248 bytes 2024-09-14T23:21:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 475259904 bytes 2024-09-14T23:21:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 482582528 bytes 2024-09-14T23:21:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 523743232 bytes 2024-09-14T23:21:53.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 596170752 bytes 2024-09-14T23:21:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 638408704 bytes 2024-09-14T23:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 736986112 bytes 2024-09-14T23:21:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 809819136 bytes 2024-09-14T23:21:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 864778240 bytes 2024-09-14T23:21:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 905281536 bytes 2024-09-14T23:21:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 917361664 bytes 2024-09-14T23:21:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 957020160 bytes 2024-09-14T23:21:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1031752704 bytes 2024-09-14T23:21:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1074608128 bytes 2024-09-14T23:21:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1096205312 bytes 2024-09-14T23:21:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1136804864 bytes 2024-09-14T23:21:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1179554816 bytes 2024-09-14T23:21:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1189007360 bytes 2024-09-14T23:21:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1233711104 bytes 2024-09-14T23:21:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1283151872 bytes 2024-09-14T23:21:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1365356544 bytes 2024-09-14T23:21:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1371408384 bytes 2024-09-14T23:21:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1407830016 bytes 2024-09-14T23:21:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1426547712 bytes 2024-09-14T23:21:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1474166784 bytes 2024-09-14T23:21:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1492315136 bytes 2024-09-14T23:21:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1529586688 bytes 2024-09-14T23:21:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1580744704 bytes 2024-09-14T23:21:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1644767232 bytes 2024-09-14T23:21:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1668025344 bytes 2024-09-14T23:21:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1686505472 bytes 2024-09-14T23:21:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1713009664 bytes 2024-09-14T23:21:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1763174400 bytes 2024-09-14T23:21:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1803548672 bytes 2024-09-14T23:21:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1881705472 bytes 2024-09-14T23:21:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1913922560 bytes 2024-09-14T23:21:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 1950306304 bytes 2024-09-14T23:21:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 1998393344 bytes 2024-09-14T23:21:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2034393088 bytes 2024-09-14T23:21:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2058969088 bytes 2024-09-14T23:21:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2086916096 bytes 2024-09-14T23:21:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2087526400 bytes 2024-09-14T23:21:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2112334848 bytes 2024-09-14T23:21:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2181000192 bytes 2024-09-14T23:21:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2194758656 bytes 2024-09-14T23:21:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2221894656 bytes 2024-09-14T23:21:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2347670528 bytes 2024-09-14T23:21:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2398268416 bytes 2024-09-14T23:21:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2398866432 bytes 2024-09-14T23:21:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2498916352 bytes 2024-09-14T23:21:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2563256320 bytes 2024-09-14T23:21:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2634860544 bytes 2024-09-14T23:21:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2708948992 bytes 2024-09-14T23:21:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2709935104 bytes 2024-09-14T23:21:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2749552640 bytes 2024-09-14T23:21:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2774344704 bytes 2024-09-14T23:21:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2822669312 bytes 2024-09-14T23:21:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 2857673728 bytes 2024-09-14T23:21:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 2941633536 bytes 2024-09-14T23:21:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 2984873984 bytes 2024-09-14T23:22:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3062501376 bytes 2024-09-14T23:22:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3138268160 bytes 2024-09-14T23:22:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3143147520 bytes 2024-09-14T23:22:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3176764416 bytes 2024-09-14T23:22:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3230697472 bytes 2024-09-14T23:22:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3253460992 bytes 2024-09-14T23:22:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3260023808 bytes 2024-09-14T23:22:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3260631040 bytes 2024-09-14T23:22:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3333605376 bytes 2024-09-14T23:22:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3351361536 bytes 2024-09-14T23:22:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3352014848 bytes 2024-09-14T23:22:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3363074048 bytes 2024-09-14T23:22:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3390302208 bytes 2024-09-14T23:22:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3448338432 bytes 2024-09-14T23:22:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3524373504 bytes 2024-09-14T23:22:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3566261248 bytes 2024-09-14T23:22:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3611110400 bytes 2024-09-14T23:22:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3630603264 bytes 2024-09-14T23:22:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3650555904 bytes 2024-09-14T23:22:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3700396032 bytes 2024-09-14T23:22:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3714409472 bytes 2024-09-14T23:22:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3776650240 bytes 2024-09-14T23:22:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 3809275904 bytes 2024-09-14T23:22:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 3822858240 bytes 2024-09-14T23:22:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:22:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:22:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:22:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:22:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-14T23:22:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:22:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:22:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:22:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:22:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:22:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T23:22:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:22:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:22:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_5 2024-09-14T23:22:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T23:22:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:22:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T23:22:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:22:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:22:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:22:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_5 present '' 2024-09-14T23:22:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:22:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:22:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:22:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:22:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:22:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T23:22:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:22:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:22:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_5 2024-09-14T23:22:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T23:22:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:22:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T23:22:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:22:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:22:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = deleted 2024-09-14T23:22:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:22:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_5 present '' 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:22:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:22:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_5 2024-09-14T23:22:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T23:22:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:22:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T23:22:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:22:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:22:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = deleted 2024-09-14T23:22:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:22:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:22:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_5 present '' 2024-09-14T23:22:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:22:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_5 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T23:22:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:22:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T23:22:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:22:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:22:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = deleted 2024-09-14T23:22:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:22:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:22:08.176 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T23:22:08.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T23:22:08.177 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T23:22:08.177 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T23:22:08.177 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:22:08.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:22:08.177+0000 7fd60e3e9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 58951) UID: 1000 2024-09-14T23:22:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_5 present '' 2024-09-14T23:22:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:22:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:22:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:22:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:22:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:22:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T23:22:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:22:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:22:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_5 2024-09-14T23:22:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T23:22:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:22:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=12aa622cdf9e 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 12aa622cdf9e 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z '' -o '' = 12aa622cdf9e 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-14T23:22:10.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:22:10.671+0000 7f39b1b45640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d65b6b8400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=189 2024-09-14T23:22:10.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:22:10.671+0000 7f39b1b45640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d65b6b8400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=191, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=bd, last_copied_object_number=128, snap_seqs={189=18446744073709551614}] 2024-09-14T23:22:14.280 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T23:22:14.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T23:22:14.281 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T23:22:14.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T23:22:14.284 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-14T23:22:15.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:22:15.158+0000 7fb3dc1db640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfc1811e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:22:15.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:22:15.158+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc414c80 [3/01b74956-f3e1-412e-bf2e-a04d6c89c901] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:22:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:22:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_5 2024-09-14T23:22:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T23:22:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:22:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T23:22:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:22:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:22:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:22:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:22:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_5 2024-09-14T23:22:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T23:22:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:22:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T23:22:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:22:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:22:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:22:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:22:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-14T23:22:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-14T23:22:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-14T23:22:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:22:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:22:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T23:22:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T23:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T23:22:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:22:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:22:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:22:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:22:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:22:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T23:22:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T23:22:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T23:22:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:22:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:22:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:22:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:22:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:22:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T23:22:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T23:22:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T23:22:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:22:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:22:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:22:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:22:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:22:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T23:22:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T23:22:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T23:22:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:22:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:22:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:22:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:22:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:22:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T23:22:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T23:22:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T23:22:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:22:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:22:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:22:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:22:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:22:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T23:22:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_5 2024-09-14T23:22:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T23:22:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T23:22:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_6 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:22:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:22:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:22:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:22:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:22:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_6 2024-09-14T23:22:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-14T23:22:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:22:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:22:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-14T23:22:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:22:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-14T23:22:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_6': 2024-09-14T23:22:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:22:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:22:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:22:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 19d878e747e0 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.19d878e747e0 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:22:47 2024 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:22:47 2024 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:22:47 2024 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 1e6466fb-8e46-4462-9a8a-caf9a11aa3eb 2024-09-14T23:22:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-14T23:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=7 2024-09-14T23:22:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:22:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:22:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 72133632 bytes 2024-09-14T23:22:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 98289664 bytes 2024-09-14T23:22:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 153332736 bytes 2024-09-14T23:22:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 165671936 bytes 2024-09-14T23:22:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 212734976 bytes 2024-09-14T23:22:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 234436608 bytes 2024-09-14T23:22:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 314183680 bytes 2024-09-14T23:22:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 354757632 bytes 2024-09-14T23:22:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 430097408 bytes 2024-09-14T23:22:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 512168960 bytes 2024-09-14T23:22:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 569457664 bytes 2024-09-14T23:22:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 642900992 bytes 2024-09-14T23:22:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 665498624 bytes 2024-09-14T23:22:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 734488576 bytes 2024-09-14T23:22:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 806268928 bytes 2024-09-14T23:22:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 832947200 bytes 2024-09-14T23:22:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 882143232 bytes 2024-09-14T23:22:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 964718592 bytes 2024-09-14T23:22:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 1065447424 bytes 2024-09-14T23:22:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 1104610304 bytes 2024-09-14T23:22:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 1116701696 bytes 2024-09-14T23:22:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1143230464 bytes 2024-09-14T23:22:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1166499840 bytes 2024-09-14T23:22:51.679 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1176922112 bytes 2024-09-14T23:22:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1238918144 bytes 2024-09-14T23:22:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1294664704 bytes 2024-09-14T23:22:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1321161728 bytes 2024-09-14T23:22:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1341603840 bytes 2024-09-14T23:22:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1352549376 bytes 2024-09-14T23:22:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1384865792 bytes 2024-09-14T23:22:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1392883712 bytes 2024-09-14T23:22:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1425024000 bytes 2024-09-14T23:22:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1452303360 bytes 2024-09-14T23:22:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1488714752 bytes 2024-09-14T23:22:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1560330240 bytes 2024-09-14T23:22:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1612198912 bytes 2024-09-14T23:22:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1675367424 bytes 2024-09-14T23:22:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1760224256 bytes 2024-09-14T23:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1826713600 bytes 2024-09-14T23:22:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1908529152 bytes 2024-09-14T23:22:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1937376256 bytes 2024-09-14T23:22:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1982942208 bytes 2024-09-14T23:22:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 2030713856 bytes 2024-09-14T23:22:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 2082929664 bytes 2024-09-14T23:22:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2109215744 bytes 2024-09-14T23:22:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2192714752 bytes 2024-09-14T23:22:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2263259136 bytes 2024-09-14T23:22:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2287629312 bytes 2024-09-14T23:22:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2323657728 bytes 2024-09-14T23:22:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2332633088 bytes 2024-09-14T23:22:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2333257728 bytes 2024-09-14T23:22:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2377341952 bytes 2024-09-14T23:22:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2421072896 bytes 2024-09-14T23:22:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2455943168 bytes 2024-09-14T23:22:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2483068928 bytes 2024-09-14T23:22:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2532167680 bytes 2024-09-14T23:22:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2561812480 bytes 2024-09-14T23:22:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:22:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:22:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-14T23:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T23:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_6 2024-09-14T23:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T23:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:22:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id=12aa8ada6947 2024-09-14T23:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:22:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_6 present 12aa8ada6947 2024-09-14T23:22:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id=12aa8ada6947 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_6 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T23:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:22:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=12aa8ada6947 2024-09-14T23:22:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 12aa8ada6947 2024-09-14T23:22:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z 12aa8ada6947 -o 12aa8ada6947 = 12aa8ada6947 2024-09-14T23:22:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:22:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:22:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:22:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-14T23:22:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:22:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:22:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:22:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-14T23:22:56.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:22:56.717+0000 7f39b1b45640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d65b6b8000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-09-14T23:22:56.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:22:56.717+0000 7f39b1b45640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d65b6b8000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=192, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-09-14T23:22:58.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:22:58.278+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfbbc3520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:22:58.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:22:58.278+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc415b80 [3/1e6466fb-8e46-4462-9a8a-caf9a11aa3eb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:23:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:23:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-14T23:23:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:23:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:23:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:23:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:23:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:23:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T23:23:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:23:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:23:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_6 2024-09-14T23:23:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T23:23:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:23:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T23:23:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:23:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:23:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:23:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:23:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_6 2024-09-14T23:23:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T23:23:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:23:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T23:23:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:23:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:23:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:23:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:23:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-14T23:23:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-14T23:23:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-14T23:23:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:23:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:23:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T23:23:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T23:23:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T23:23:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:23:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:23:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:23:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:23:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:23:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T23:23:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T23:23:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T23:23:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:23:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:23:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:23:03.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:23:03.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:23:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T23:23:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T23:23:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T23:23:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:23:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:23:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:23:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:23:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:23:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T23:23:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T23:23:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T23:23:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:23:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:23:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:23:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:23:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:23:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T23:23:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T23:23:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T23:23:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:23:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:23:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:23:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:23:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:23:20.278 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4753276911154676) > weight (0.25) 2024-09-14T23:23:20.279 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-14T23:23:20.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-14T23:23:20.280 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T23:23:20.280 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-14T23:23:20.280 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:23:20.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:23:20.280+0000 7f39bf360640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 49566) UID: 1000 2024-09-14T23:23:26.381 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-14T23:23:26.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T23:23:26.382 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T23:23:26.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T23:23:26.385 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-14T23:23:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T23:23:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_6 2024-09-14T23:23:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T23:23:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T23:23:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:23:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_7 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:23:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:23:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_7 2024-09-14T23:23:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-14T23:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:23:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-14T23:23:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:23:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-14T23:23:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_7': 2024-09-14T23:23:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:23:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:23:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:23:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1a141d01afb5 2024-09-14T23:23:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1a141d01afb5 2024-09-14T23:23:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:23:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:23:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:23:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:23:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:23:33 2024 2024-09-14T23:23:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:23:33 2024 2024-09-14T23:23:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:23:33 2024 2024-09-14T23:23:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:23:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:23:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 8a97c69c-e8f4-44c5-8a70-04fc5568b004 2024-09-14T23:23:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:23:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-14T23:23:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:23:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:23:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=13 2024-09-14T23:23:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:23:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:23:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 82151424 bytes 2024-09-14T23:23:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 123188224 bytes 2024-09-14T23:23:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 152178688 bytes 2024-09-14T23:23:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 221031424 bytes 2024-09-14T23:23:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 282499072 bytes 2024-09-14T23:23:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 307640320 bytes 2024-09-14T23:23:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 336382976 bytes 2024-09-14T23:23:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 372418560 bytes 2024-09-14T23:23:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 410929152 bytes 2024-09-14T23:23:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 426587136 bytes 2024-09-14T23:23:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 449497088 bytes 2024-09-14T23:23:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 473726976 bytes 2024-09-14T23:23:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 555266048 bytes 2024-09-14T23:23:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 588733440 bytes 2024-09-14T23:23:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 609618944 bytes 2024-09-14T23:23:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 610248704 bytes 2024-09-14T23:23:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 654146560 bytes 2024-09-14T23:23:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 674318336 bytes 2024-09-14T23:23:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 705225728 bytes 2024-09-14T23:23:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 734300160 bytes 2024-09-14T23:23:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 794022912 bytes 2024-09-14T23:23:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 824283136 bytes 2024-09-14T23:23:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 900093952 bytes 2024-09-14T23:23:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 937605120 bytes 2024-09-14T23:23:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 968772608 bytes 2024-09-14T23:23:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1005089792 bytes 2024-09-14T23:23:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1047312384 bytes 2024-09-14T23:23:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1074963456 bytes 2024-09-14T23:23:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1088947200 bytes 2024-09-14T23:23:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1104760832 bytes 2024-09-14T23:23:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1215906816 bytes 2024-09-14T23:23:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1228739584 bytes 2024-09-14T23:23:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1310410752 bytes 2024-09-14T23:23:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1337703424 bytes 2024-09-14T23:23:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1379091456 bytes 2024-09-14T23:23:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1400823808 bytes 2024-09-14T23:23:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1425430528 bytes 2024-09-14T23:23:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1463870464 bytes 2024-09-14T23:23:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1517412352 bytes 2024-09-14T23:23:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1619885056 bytes 2024-09-14T23:23:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1623518208 bytes 2024-09-14T23:23:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1670726656 bytes 2024-09-14T23:23:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1739028480 bytes 2024-09-14T23:23:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1829389312 bytes 2024-09-14T23:23:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1931779072 bytes 2024-09-14T23:23:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1948129280 bytes 2024-09-14T23:23:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2009356288 bytes 2024-09-14T23:23:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2067367936 bytes 2024-09-14T23:23:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2128776192 bytes 2024-09-14T23:23:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2201829376 bytes 2024-09-14T23:23:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2211015680 bytes 2024-09-14T23:23:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2223325184 bytes 2024-09-14T23:23:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2279911424 bytes 2024-09-14T23:23:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2352657408 bytes 2024-09-14T23:23:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2377311232 bytes 2024-09-14T23:23:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2420442112 bytes 2024-09-14T23:23:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2430918656 bytes 2024-09-14T23:23:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2469429248 bytes 2024-09-14T23:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2532501504 bytes 2024-09-14T23:23:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2610875392 bytes 2024-09-14T23:23:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2651279360 bytes 2024-09-14T23:23:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2703774720 bytes 2024-09-14T23:23:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2722509824 bytes 2024-09-14T23:23:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2791709696 bytes 2024-09-14T23:23:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2810997760 bytes 2024-09-14T23:23:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2898922496 bytes 2024-09-14T23:23:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2930140160 bytes 2024-09-14T23:23:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2991912960 bytes 2024-09-14T23:23:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3068455936 bytes 2024-09-14T23:23:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3101489152 bytes 2024-09-14T23:23:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3104175104 bytes 2024-09-14T23:23:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3147506688 bytes 2024-09-14T23:23:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3185527808 bytes 2024-09-14T23:23:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3186140160 bytes 2024-09-14T23:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3225245696 bytes 2024-09-14T23:23:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3287775232 bytes 2024-09-14T23:23:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3344047104 bytes 2024-09-14T23:23:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3363346432 bytes 2024-09-14T23:23:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3410536448 bytes 2024-09-14T23:23:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3461900288 bytes 2024-09-14T23:23:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3487635456 bytes 2024-09-14T23:23:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3522581504 bytes 2024-09-14T23:23:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3542749184 bytes 2024-09-14T23:23:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3581111296 bytes 2024-09-14T23:23:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3590864896 bytes 2024-09-14T23:23:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3633076224 bytes 2024-09-14T23:23:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3647668224 bytes 2024-09-14T23:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3726621696 bytes 2024-09-14T23:23:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3756506112 bytes 2024-09-14T23:23:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3765186560 bytes 2024-09-14T23:23:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 3775987712 bytes 2024-09-14T23:23:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 3820225536 bytes 2024-09-14T23:23:45.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 3857861632 bytes 2024-09-14T23:23:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 3919662080 bytes 2024-09-14T23:23:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 3957635072 bytes 2024-09-14T23:23:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4020792320 bytes 2024-09-14T23:23:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4053468160 bytes 2024-09-14T23:23:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4078987264 bytes 2024-09-14T23:23:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4163948544 bytes 2024-09-14T23:23:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4227537920 bytes 2024-09-14T23:23:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4286384128 bytes 2024-09-14T23:23:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4321705984 bytes 2024-09-14T23:23:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4369532928 bytes 2024-09-14T23:23:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4398956544 bytes 2024-09-14T23:23:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4411249664 bytes 2024-09-14T23:23:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4425013248 bytes 2024-09-14T23:23:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4532474880 bytes 2024-09-14T23:23:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4552160256 bytes 2024-09-14T23:23:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 4575239168 bytes 2024-09-14T23:23:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:11000 IOs, 4606489600 bytes 2024-09-14T23:23:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:23:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:23:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:23:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:23:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-14T23:23:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:23:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:23:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:23:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:23:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:23:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:23:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T23:23:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:23:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:23:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_7 2024-09-14T23:23:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T23:23:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:23:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T23:23:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:23:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:23:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_7 present '' 2024-09-14T23:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:23:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:23:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:23:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:23:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T23:23:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:23:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:23:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_7 2024-09-14T23:23:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T23:23:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:23:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T23:23:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:23:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:23:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = deleted 2024-09-14T23:23:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:23:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_7 present '' 2024-09-14T23:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T23:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_7 2024-09-14T23:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T23:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:23:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T23:23:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:23:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:23:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = deleted 2024-09-14T23:23:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:23:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:23:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_7 present '' 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_7 2024-09-14T23:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T23:23:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T23:23:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:23:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:23:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = deleted 2024-09-14T23:23:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:23:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:23:53.285 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T23:23:53.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T23:23:53.285 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-14T23:23:53.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:23:53.285+0000 7f6f8f43a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 72063) UID: 1000 2024-09-14T23:23:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_7 present '' 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:23:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_7 2024-09-14T23:23:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T23:23:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:23:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=165e80cdbd31 2024-09-14T23:23:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 165e80cdbd31 2024-09-14T23:23:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z '' -o '' = 165e80cdbd31 2024-09-14T23:23:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:23:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:23:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:23:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-14T23:23:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:23:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:23:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:23:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-14T23:23:55.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:23:55.496+0000 7f577fd6c640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e49a9d8800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=191 2024-09-14T23:23:55.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:23:55.496+0000 7f577fd6c640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e49a9d8800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=193, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=bf, last_copied_object_number=128, snap_seqs={191=18446744073709551614}] 2024-09-14T23:23:59.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:23:59.510+0000 7fb3db9da640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558cfc180d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T23:23:59.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:23:59.510+0000 7fb3e29e8640 -1 rbd::mirror::ImageReplayer: 0x558cfc415b80 [3/8a97c69c-e8f4-44c5-8a70-04fc5568b004] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T23:24:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:24:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-14T23:24:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:24:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:24:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:24:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:24:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:24:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:24:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T23:24:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:24:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:24:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_7 2024-09-14T23:24:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T23:24:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T23:24:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:24:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:24:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:24:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-14T23:24:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:24:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:24:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:24:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:24:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:24:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:24:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T23:24:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:24:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:24:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_7 2024-09-14T23:24:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T23:24:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:24:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T23:24:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:24:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:24:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:24:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:24:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-14T23:24:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:24:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-14T23:24:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-14T23:24:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:24:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:24:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T23:24:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:24:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T23:24:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T23:24:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:24:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:24:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:24:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:24:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T23:24:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:24:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T23:24:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T23:24:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:24:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:24:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:24:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:24:02.286 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T23:24:02.286 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:24:02.286 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T23:24:02.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T23:24:02.287 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T23:24:02.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T23:24:02.290 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-14T23:24:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T23:24:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:24:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T23:24:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T23:24:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:24:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:24:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:24:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:24:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T23:24:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:24:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T23:24:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T23:24:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:24:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:24:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:24:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:24:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T23:24:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_7 2024-09-14T23:24:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T23:24:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T23:24:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:24:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:24:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:24:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_8 2024-09-14T23:24:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:24:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_8 2024-09-14T23:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-14T23:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-14T23:24:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:24:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-14T23:24:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_8': 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1a465ab710b9 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1a465ab710b9 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:24:16 2024 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:24:16 2024 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:24:16 2024 2024-09-14T23:24:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:24:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:24:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 87201032-7588-43eb-a991-e20525050837 2024-09-14T23:24:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:24:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-14T23:24:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:24:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=13 2024-09-14T23:24:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:24:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:24:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 36170752 bytes 2024-09-14T23:24:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 76428288 bytes 2024-09-14T23:24:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 129575936 bytes 2024-09-14T23:24:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 175833088 bytes 2024-09-14T23:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 246308864 bytes 2024-09-14T23:24:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 286238720 bytes 2024-09-14T23:24:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 311552000 bytes 2024-09-14T23:24:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 355733504 bytes 2024-09-14T23:24:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 363508736 bytes 2024-09-14T23:24:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 436885504 bytes 2024-09-14T23:24:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 553501696 bytes 2024-09-14T23:24:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 590159872 bytes 2024-09-14T23:24:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 590756864 bytes 2024-09-14T23:24:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 630222848 bytes 2024-09-14T23:24:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 673859584 bytes 2024-09-14T23:24:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 731487232 bytes 2024-09-14T23:24:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 768652288 bytes 2024-09-14T23:24:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 839115776 bytes 2024-09-14T23:24:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 889003008 bytes 2024-09-14T23:24:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 962743296 bytes 2024-09-14T23:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 988622848 bytes 2024-09-14T23:24:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 1038647296 bytes 2024-09-14T23:24:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1070055424 bytes 2024-09-14T23:24:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1147151360 bytes 2024-09-14T23:24:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1202900992 bytes 2024-09-14T23:24:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1223500800 bytes 2024-09-14T23:24:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1288097792 bytes 2024-09-14T23:24:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1307378688 bytes 2024-09-14T23:24:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1372951552 bytes 2024-09-14T23:24:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1399473152 bytes 2024-09-14T23:24:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1442939904 bytes 2024-09-14T23:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1446148096 bytes 2024-09-14T23:24:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1482848256 bytes 2024-09-14T23:24:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1557301248 bytes 2024-09-14T23:24:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1587589120 bytes 2024-09-14T23:24:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1638312960 bytes 2024-09-14T23:24:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1639784448 bytes 2024-09-14T23:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1761201152 bytes 2024-09-14T23:24:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1839550464 bytes 2024-09-14T23:24:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1872950272 bytes 2024-09-14T23:24:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1895699456 bytes 2024-09-14T23:24:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1982868480 bytes 2024-09-14T23:24:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 2050226176 bytes 2024-09-14T23:24:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 2095025152 bytes 2024-09-14T23:24:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 2134838272 bytes 2024-09-14T23:24:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 2150805504 bytes 2024-09-14T23:24:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2175893504 bytes 2024-09-14T23:24:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2218011648 bytes 2024-09-14T23:24:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2228382720 bytes 2024-09-14T23:24:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2250587136 bytes 2024-09-14T23:24:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2251892736 bytes 2024-09-14T23:24:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2309607424 bytes 2024-09-14T23:24:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2370843648 bytes 2024-09-14T23:24:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2439083008 bytes 2024-09-14T23:24:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2501405696 bytes 2024-09-14T23:24:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2568386560 bytes 2024-09-14T23:24:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2575459328 bytes 2024-09-14T23:24:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2622452736 bytes 2024-09-14T23:24:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2669634560 bytes 2024-09-14T23:24:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2735502336 bytes 2024-09-14T23:24:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2761776128 bytes 2024-09-14T23:24:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2790973440 bytes 2024-09-14T23:24:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2871641088 bytes 2024-09-14T23:24:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2917803008 bytes 2024-09-14T23:24:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2921596928 bytes 2024-09-14T23:24:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2942699520 bytes 2024-09-14T23:24:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 3000474624 bytes 2024-09-14T23:24:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 3057159168 bytes 2024-09-14T23:24:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 3073053696 bytes 2024-09-14T23:24:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7000 IOs, 3098650624 bytes 2024-09-14T23:24:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7100 IOs, 3107309568 bytes 2024-09-14T23:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7200 IOs, 3153847296 bytes 2024-09-14T23:24:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7300 IOs, 3190332416 bytes 2024-09-14T23:24:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7400 IOs, 3194416128 bytes 2024-09-14T23:24:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7500 IOs, 3313893376 bytes 2024-09-14T23:24:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7600 IOs, 3366648832 bytes 2024-09-14T23:24:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7700 IOs, 3370519552 bytes 2024-09-14T23:24:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7800 IOs, 3418227712 bytes 2024-09-14T23:24:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:7900 IOs, 3514018816 bytes 2024-09-14T23:24:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8000 IOs, 3551403008 bytes 2024-09-14T23:24:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8100 IOs, 3606886400 bytes 2024-09-14T23:24:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8200 IOs, 3659407360 bytes 2024-09-14T23:24:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8300 IOs, 3722753024 bytes 2024-09-14T23:24:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8400 IOs, 3781952512 bytes 2024-09-14T23:24:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8500 IOs, 3847164928 bytes 2024-09-14T23:24:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8600 IOs, 3851431936 bytes 2024-09-14T23:24:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8700 IOs, 3875203072 bytes 2024-09-14T23:24:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8800 IOs, 3900383232 bytes 2024-09-14T23:24:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:8900 IOs, 3900996608 bytes 2024-09-14T23:24:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9000 IOs, 3962959872 bytes 2024-09-14T23:24:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9100 IOs, 4004558848 bytes 2024-09-14T23:24:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9200 IOs, 4026961920 bytes 2024-09-14T23:24:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9300 IOs, 4087949312 bytes 2024-09-14T23:24:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9400 IOs, 4136353792 bytes 2024-09-14T23:24:27.908 INFO:tasks.ceph.cluster2.osd.0.smithi089.stderr:2024-09-14T23:24:27.906+0000 7fc768a4a640 -1 reset not still connected to 0x56493df74680 2024-09-14T23:24:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9500 IOs, 4183108608 bytes 2024-09-14T23:24:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9600 IOs, 4260558848 bytes 2024-09-14T23:24:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9700 IOs, 4319219712 bytes 2024-09-14T23:24:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9800 IOs, 4371607552 bytes 2024-09-14T23:24:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:9900 IOs, 4430045184 bytes 2024-09-14T23:24:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10000 IOs, 4504501248 bytes 2024-09-14T23:24:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10100 IOs, 4567266304 bytes 2024-09-14T23:24:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10200 IOs, 4578584576 bytes 2024-09-14T23:24:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10300 IOs, 4634092544 bytes 2024-09-14T23:24:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10400 IOs, 4655093760 bytes 2024-09-14T23:24:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10500 IOs, 4688521216 bytes 2024-09-14T23:24:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10600 IOs, 4746680320 bytes 2024-09-14T23:24:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10700 IOs, 4786958336 bytes 2024-09-14T23:24:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10800 IOs, 4808566784 bytes 2024-09-14T23:24:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:10900 IOs, 4813275136 bytes 2024-09-14T23:24:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:24:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:24:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:24:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:24:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-14T23:24:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:24:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_8 2024-09-14T23:24:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T23:24:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:24:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id=165ef3020e50 2024-09-14T23:24:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:24:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:24:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_8 present 165ef3020e50 2024-09-14T23:24:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:24:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:24:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id=165ef3020e50 2024-09-14T23:24:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:24:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:24:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T23:24:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:24:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:24:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_8 2024-09-14T23:24:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T23:24:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:24:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=165ef3020e50 2024-09-14T23:24:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 165ef3020e50 2024-09-14T23:24:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z 165ef3020e50 -o 165ef3020e50 = 165ef3020e50 2024-09-14T23:24:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:24:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:24:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:24:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-14T23:24:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-14T23:24:30.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:24:30.669+0000 7f577fd6c640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e49aa9f400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=192 2024-09-14T23:24:30.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:24:30.669+0000 7f577fd6c640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e49aa9f400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=194, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=c0, last_copied_object_number=128, snap_seqs={192=18446744073709551614}] 2024-09-14T23:24:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-14T23:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:24:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:24:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:24:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:24:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T23:24:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:24:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:24:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_8 2024-09-14T23:24:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T23:24:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:24:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T23:24:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:24:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:24:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:24:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-14T23:24:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:24:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:24:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:24:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:24:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:24:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T23:24:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:24:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:24:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_8 2024-09-14T23:24:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T23:24:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T23:24:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:24:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:24:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:24:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:24:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-14T23:24:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-14T23:24:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-14T23:24:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:24:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T23:24:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T23:24:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T23:24:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:24:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:24:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:24:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T23:24:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T23:24:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T23:24:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:24:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:24:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:24:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:24:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T23:24:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T23:24:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T23:24:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:24:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:24:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:24:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:24:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T23:24:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T23:24:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T23:24:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:24:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:24:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:24:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:24:43.386 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9243602193800025) > weight (0.25) 2024-09-14T23:24:43.386 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-14T23:24:43.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-14T23:24:43.387 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-14T23:24:43.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:24:43.387+0000 7f22acec8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 72901) UID: 1000 2024-09-14T23:24:47.388 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-14T23:24:47.388 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:24:47.388 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-14T23:24:47.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T23:24:47.389 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T23:24:47.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T23:24:47.392 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-14T23:24:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T23:24:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:24:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:24:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:24:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T23:24:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T23:24:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:24:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:24:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:24:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:24:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:25:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T23:25:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_8 2024-09-14T23:25:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T23:25:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T23:25:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_9 2024-09-14T23:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-14T23:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:25:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:25:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:25:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:25:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:25:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_9 2024-09-14T23:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-14T23:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:25:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-14T23:25:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:25:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_9': 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1a8278cbc105 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1a8278cbc105 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:25:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:25:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:25:07 2024 2024-09-14T23:25:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:25:07 2024 2024-09-14T23:25:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:25:07 2024 2024-09-14T23:25:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:25:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:25:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 5aaf8271-7b03-495f-b374-ca75df46c288 2024-09-14T23:25:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:25:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-14T23:25:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:25:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=8 2024-09-14T23:25:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:25:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:25:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 51527680 bytes 2024-09-14T23:25:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 212302848 bytes 2024-09-14T23:25:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 238312448 bytes 2024-09-14T23:25:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 299272192 bytes 2024-09-14T23:25:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 316644352 bytes 2024-09-14T23:25:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 372503552 bytes 2024-09-14T23:25:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 387033088 bytes 2024-09-14T23:25:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 424290304 bytes 2024-09-14T23:25:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 437528576 bytes 2024-09-14T23:25:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 470245376 bytes 2024-09-14T23:25:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 589888512 bytes 2024-09-14T23:25:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 620402688 bytes 2024-09-14T23:25:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 689998848 bytes 2024-09-14T23:25:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 699060224 bytes 2024-09-14T23:25:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 716819456 bytes 2024-09-14T23:25:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 737094656 bytes 2024-09-14T23:25:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 780687360 bytes 2024-09-14T23:25:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 791043072 bytes 2024-09-14T23:25:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 849329152 bytes 2024-09-14T23:25:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 906605568 bytes 2024-09-14T23:25:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 920793088 bytes 2024-09-14T23:25:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 971969536 bytes 2024-09-14T23:25:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 1020456960 bytes 2024-09-14T23:25:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 1026350080 bytes 2024-09-14T23:25:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 1030068224 bytes 2024-09-14T23:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1030673408 bytes 2024-09-14T23:25:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1051617280 bytes 2024-09-14T23:25:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1123926016 bytes 2024-09-14T23:25:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1177636864 bytes 2024-09-14T23:25:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1211154432 bytes 2024-09-14T23:25:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1228267520 bytes 2024-09-14T23:25:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1332921344 bytes 2024-09-14T23:25:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1358944256 bytes 2024-09-14T23:25:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1404998656 bytes 2024-09-14T23:25:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1488197632 bytes 2024-09-14T23:25:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1558584320 bytes 2024-09-14T23:25:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1559209984 bytes 2024-09-14T23:25:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1591758848 bytes 2024-09-14T23:25:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1645600768 bytes 2024-09-14T23:25:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1725284352 bytes 2024-09-14T23:25:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1752174592 bytes 2024-09-14T23:25:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1798821888 bytes 2024-09-14T23:25:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1865244672 bytes 2024-09-14T23:25:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1910226944 bytes 2024-09-14T23:25:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1950374912 bytes 2024-09-14T23:25:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1978116096 bytes 2024-09-14T23:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 2007004160 bytes 2024-09-14T23:25:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 2065782784 bytes 2024-09-14T23:25:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4900 IOs, 2084401152 bytes 2024-09-14T23:25:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5000 IOs, 2129022976 bytes 2024-09-14T23:25:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5100 IOs, 2153270272 bytes 2024-09-14T23:25:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5200 IOs, 2234006528 bytes 2024-09-14T23:25:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5300 IOs, 2280203264 bytes 2024-09-14T23:25:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5400 IOs, 2339752960 bytes 2024-09-14T23:25:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5500 IOs, 2399244288 bytes 2024-09-14T23:25:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5600 IOs, 2419902464 bytes 2024-09-14T23:25:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5700 IOs, 2455382016 bytes 2024-09-14T23:25:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5800 IOs, 2458591232 bytes 2024-09-14T23:25:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:5900 IOs, 2520033280 bytes 2024-09-14T23:25:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6000 IOs, 2607544320 bytes 2024-09-14T23:25:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6100 IOs, 2646323200 bytes 2024-09-14T23:25:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6200 IOs, 2672372736 bytes 2024-09-14T23:25:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6300 IOs, 2672984064 bytes 2024-09-14T23:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6400 IOs, 2698763264 bytes 2024-09-14T23:25:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6500 IOs, 2737790976 bytes 2024-09-14T23:25:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6600 IOs, 2761183232 bytes 2024-09-14T23:25:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6700 IOs, 2798745600 bytes 2024-09-14T23:25:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6800 IOs, 2815691776 bytes 2024-09-14T23:25:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:6900 IOs, 2872343552 bytes 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:25:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:25:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T23:25:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:25:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:25:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_9 2024-09-14T23:25:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T23:25:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:25:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id=165e2525dda1 2024-09-14T23:25:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:25:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:25:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_9 present 165e2525dda1 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id=165e2525dda1 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:25:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_9 2024-09-14T23:25:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T23:25:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:25:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=165e2525dda1 2024-09-14T23:25:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 165e2525dda1 2024-09-14T23:25:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z 165e2525dda1 -o 165e2525dda1 = 165e2525dda1 2024-09-14T23:25:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:25:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:25:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:25:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-14T23:25:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-14T23:25:17.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:25:17.663+0000 7f578056d640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e49aa80400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=193 2024-09-14T23:25:17.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:25:17.663+0000 7f578056d640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e49aa80400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=195, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=c1, last_copied_object_number=128, snap_seqs={193=18446744073709551614}] 2024-09-14T23:25:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T23:25:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:25:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:25:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_9 2024-09-14T23:25:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T23:25:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:25:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T23:25:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:25:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:25:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:25:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_9 2024-09-14T23:25:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T23:25:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:25:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T23:25:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:25:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:25:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:25:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:25:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-14T23:25:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-14T23:25:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-14T23:25:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:25:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:25:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T23:25:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T23:25:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T23:25:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:25:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:25:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:25:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:25:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:25:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T23:25:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T23:25:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T23:25:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:25:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:25:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:25:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:25:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:25:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T23:25:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T23:25:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T23:25:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:25:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:25:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:25:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:25:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:25:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T23:25:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T23:25:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T23:25:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:25:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:25:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:25:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:25:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:25:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T23:25:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T23:25:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T23:25:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:25:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:25:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:25:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:25:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:25:48.291 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T23:25:48.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T23:25:48.292 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-14T23:25:48.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:25:48.292+0000 7f28bcf6f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 73431) UID: 1000 2024-09-14T23:25:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T23:25:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_9 2024-09-14T23:25:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T23:25:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T23:25:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:25:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:25:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for i in `seq 1 10` 2024-09-14T23:25:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image=image_10 2024-09-14T23:25:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-14T23:25:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:25:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:25:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:25:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:25:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local size=128 2024-09-14T23:25:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -n 512M ']' 2024-09-14T23:25:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ size=512M 2024-09-14T23:25:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ shift 2024-09-14T23:25:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_10 2024-09-14T23:25:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-14T23:25:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:25:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local mode=snapshot 2024-09-14T23:25:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-14T23:25:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring enabled 2024-09-14T23:25:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:rbd image 'image_10': 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: size 512 MiB in 128 objects 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: order 22 (4 MiB objects) 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: snapshot_count: 1 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: id: 1ab5bca3989e 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: block_name_prefix: rbd_data.1ab5bca3989e 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: format: 2 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: features: layering 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: op_features: 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: flags: 2024-09-14T23:25:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: create_timestamp: Sat Sep 14 23:25:54 2024 2024-09-14T23:25:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: access_timestamp: Sat Sep 14 23:25:54 2024 2024-09-14T23:25:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: modify_timestamp: Sat Sep 14 23:25:54 2024 2024-09-14T23:25:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring state: enabled 2024-09-14T23:25:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring mode: snapshot 2024-09-14T23:25:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring global id: 146a4e54-73e0-4297-b9d3-f93b45d54e38 2024-09-14T23:25:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout: mirroring primary: true 2024-09-14T23:25:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-14T23:25:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:25:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:25:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:25:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T23:25:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local duration=6 2024-09-14T23:25:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:25:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-14T23:25:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:100 IOs, 615424 bytes 2024-09-14T23:25:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:200 IOs, 53977088 bytes 2024-09-14T23:25:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:300 IOs, 77508608 bytes 2024-09-14T23:25:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:400 IOs, 137697280 bytes 2024-09-14T23:25:56.293 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T23:25:56.293 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:25:56.294 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T23:25:56.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T23:25:56.294 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T23:25:56.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T23:25:56.297 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-09-14T23:25:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:500 IOs, 192571392 bytes 2024-09-14T23:25:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:600 IOs, 232957952 bytes 2024-09-14T23:25:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:700 IOs, 277983232 bytes 2024-09-14T23:25:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:800 IOs, 281617408 bytes 2024-09-14T23:25:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:900 IOs, 326676480 bytes 2024-09-14T23:25:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1000 IOs, 349295616 bytes 2024-09-14T23:25:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1100 IOs, 402704384 bytes 2024-09-14T23:25:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1200 IOs, 418595840 bytes 2024-09-14T23:25:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1300 IOs, 487263232 bytes 2024-09-14T23:25:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1400 IOs, 536421376 bytes 2024-09-14T23:25:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1500 IOs, 576384000 bytes 2024-09-14T23:25:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1600 IOs, 616294400 bytes 2024-09-14T23:25:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1700 IOs, 625054720 bytes 2024-09-14T23:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1800 IOs, 629814272 bytes 2024-09-14T23:25:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:1900 IOs, 680764416 bytes 2024-09-14T23:25:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2000 IOs, 737271808 bytes 2024-09-14T23:25:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2100 IOs, 772792320 bytes 2024-09-14T23:25:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2200 IOs, 804753408 bytes 2024-09-14T23:25:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2300 IOs, 877623296 bytes 2024-09-14T23:25:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2400 IOs, 926683136 bytes 2024-09-14T23:25:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2500 IOs, 997425152 bytes 2024-09-14T23:25:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2600 IOs, 1044523008 bytes 2024-09-14T23:25:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2700 IOs, 1116880896 bytes 2024-09-14T23:25:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2800 IOs, 1125800960 bytes 2024-09-14T23:25:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:2900 IOs, 1147495424 bytes 2024-09-14T23:25:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3000 IOs, 1181774848 bytes 2024-09-14T23:25:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3100 IOs, 1201110016 bytes 2024-09-14T23:25:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3200 IOs, 1243077632 bytes 2024-09-14T23:25:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3300 IOs, 1265857536 bytes 2024-09-14T23:25:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3400 IOs, 1296570368 bytes 2024-09-14T23:25:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3500 IOs, 1336423424 bytes 2024-09-14T23:26:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3600 IOs, 1397621760 bytes 2024-09-14T23:26:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3700 IOs, 1420870656 bytes 2024-09-14T23:26:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3800 IOs, 1455646720 bytes 2024-09-14T23:26:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:3900 IOs, 1545137152 bytes 2024-09-14T23:26:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4000 IOs, 1568167936 bytes 2024-09-14T23:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4100 IOs, 1624973312 bytes 2024-09-14T23:26:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4200 IOs, 1642884096 bytes 2024-09-14T23:26:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4300 IOs, 1672279040 bytes 2024-09-14T23:26:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4400 IOs, 1723257856 bytes 2024-09-14T23:26:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4500 IOs, 1743783936 bytes 2024-09-14T23:26:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4600 IOs, 1789214720 bytes 2024-09-14T23:26:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4700 IOs, 1876870144 bytes 2024-09-14T23:26:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:4800 IOs, 1911257088 bytes 2024-09-14T23:26:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=124 2024-09-14T23:26:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:26:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T23:26:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-14T23:26:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:26:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=present 2024-09-14T23:26:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:26:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:26:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:26:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T23:26:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_10 2024-09-14T23:26:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T23:26:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:26:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id=165efd5d7d35 2024-09-14T23:26:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:26:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:26:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_10 present 165efd5d7d35 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=present 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id=165efd5d7d35 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_10 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T23:26:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:26:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id=165efd5d7d35 2024-09-14T23:26:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 165efd5d7d35 2024-09-14T23:26:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -z 165efd5d7d35 -o 165efd5d7d35 = 165efd5d7d35 2024-09-14T23:26:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_state=present 2024-09-14T23:26:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test present = present 2024-09-14T23:26:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-14T23:26:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-14T23:26:02.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:02.715+0000 7f578056d640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e49b02c400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=194 2024-09-14T23:26:02.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:02.715+0000 7f578056d640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e49b02c400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=196, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=a4ac212d-4b54-4fda-99c8-ab004b275d38, primary_snap_id=c2, last_copied_object_number=128, snap_seqs={194=18446744073709551614}] 2024-09-14T23:26:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:Mirroring disabled 2024-09-14T23:26:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local state=deleted 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local s 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_10 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T23:26:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:26:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T23:26:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ image_id= 2024-09-14T23:26:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T23:26:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0.1 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local test_state=deleted 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image_id= 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_state=deleted 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local current_image_id 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T23:26:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local image=image_10 2024-09-14T23:26:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T23:26:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T23:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T23:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ current_image_id= 2024-09-14T23:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n '' 2024-09-14T23:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test deleted = deleted 2024-09-14T23:26:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-14T23:26:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-14T23:26:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-14T23:26:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:26:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:26:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T23:26:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T23:26:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T23:26:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:26:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:26:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:26:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:26:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 1 2024-09-14T23:26:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T23:26:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T23:26:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T23:26:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:26:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:26:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:26:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:26:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 2 2024-09-14T23:26:09.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3954986493692443) > weight (0.25) 2024-09-14T23:26:09.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.29780615687140444) > weight (0.25) 2024-09-14T23:26:09.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.43789185217677984) > weight (0.25) 2024-09-14T23:26:09.394 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-14T23:26:09.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-14T23:26:09.394 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-14T23:26:09.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:09.394+0000 7fb6dd130640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 71699) UID: 1000 2024-09-14T23:26:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T23:26:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T23:26:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T23:26:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:26:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:26:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:26:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:26:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 4 2024-09-14T23:26:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T23:26:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T23:26:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T23:26:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:26:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:26:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:26:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:26:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 8 2024-09-14T23:26:15.395 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-14T23:26:15.395 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T23:26:15.395 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-14T23:26:15.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T23:26:15.395 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T23:26:15.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T23:26:15.398 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-14T23:26:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T23:26:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T23:26:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T23:26:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 0% complete...failed. 2024-09-14T23:26:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:rbd: error: image still has watchers 2024-09-14T23:26:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-14T23:26:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T23:26:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 16 2024-09-14T23:26:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T23:26:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local image=image_10 2024-09-14T23:26:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T23:26:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T23:26:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T23:26:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-14T23:26:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ tee -a /tmp/tmp.OxLGv2e3xU/rbd-mirror.test.log 2024-09-14T23:26:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ date '+%F %T' 2024-09-14T23:26:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo 2024-09-14 23:26:39 TEST: check if removed 'images'\''' OMAP are removed 2024-09-14T23:26:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:2024-09-14 23:26:39 TEST: check if removed images' OMAP are removed 2024-09-14T23:26:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-14T23:26:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:26:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-14T23:26:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:26:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local obj_name=rbd_mirroring 2024-09-14T23:26:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local filter=status_global 2024-09-14T23:26:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T23:26:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirroring 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local filter=status_global 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirroring 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -c status_global 2024-09-14T23:26:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-14T23:26:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 = 0 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local obj_name=rbd_mirroring 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local filter=image_ 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T23:26:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirroring 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local filter=image_ 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirroring 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-14T23:26:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -c image_ 2024-09-14T23:26:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 = 0 2024-09-14T23:26:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T23:26:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T23:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-14T23:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local obj_name=rbd_mirror_leader 2024-09-14T23:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local filter=image_map 2024-09-14T23:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T23:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-14T23:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T23:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local filter=image_map 2024-09-14T23:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-14T23:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster1 2024-09-14T23:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T23:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T23:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -c image_map 2024-09-14T23:26:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 = 0 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local obj_name=rbd_mirroring 2024-09-14T23:26:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local filter=status_global 2024-09-14T23:26:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T23:26:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:26:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster2 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirroring 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local filter=status_global 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster2 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirroring 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -c status_global 2024-09-14T23:26:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-14T23:26:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 = 0 2024-09-14T23:26:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T23:26:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:26:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T23:26:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-14T23:26:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local obj_name=rbd_mirroring 2024-09-14T23:26:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local filter=image_ 2024-09-14T23:26:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T23:26:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-14T23:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster2 2024-09-14T23:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirroring 2024-09-14T23:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local filter=image_ 2024-09-14T23:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-14T23:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster2 2024-09-14T23:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirroring 2024-09-14T23:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-14T23:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -c image_ 2024-09-14T23:26:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 = 0 2024-09-14T23:26:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T23:26:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:26:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T23:26:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-14T23:26:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local pool=mirror 2024-09-14T23:26:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local obj_name=rbd_mirror_leader 2024-09-14T23:26:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local filter=image_map 2024-09-14T23:26:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T23:26:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ sleep 0 2024-09-14T23:26:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:26:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-14T23:26:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster2 2024-09-14T23:26:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T23:26:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local filter=image_map 2024-09-14T23:26:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-14T23:26:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local cluster=cluster2 2024-09-14T23:26:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local pool=mirror 2024-09-14T23:26:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T23:26:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T23:26:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ grep -c image_map 2024-09-14T23:26:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test 0 = 0 2024-09-14T23:26:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ error_code=0 2024-09-14T23:26:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set -e 2024-09-14T23:26:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T23:26:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 0 2024-09-14T23:26:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ cleanup 0 2024-09-14T23:26:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local error_code=0 2024-09-14T23:26:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ set +e 2024-09-14T23:26:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -ne 0 ']' 2024-09-14T23:26:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -z '' ']' 2024-09-14T23:26:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster instance 2024-09-14T23:26:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T23:26:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-14T23:26:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:pool 'mirror' does not exist 2024-09-14T23:26:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T23:26:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-14T23:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:pool 'mirror' does not exist 2024-09-14T23:26:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T23:26:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-14T23:26:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:pool 'mirror_parent' does not exist 2024-09-14T23:26:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ CEPH_ARGS= 2024-09-14T23:26:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-14T23:26:43.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3e29e8640 -1 rbd::mirror::LeaderWatcher: 0x558cfbeb2c00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-14T23:26:43.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3dc1db640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558cfcacc900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T23:26:43.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3dc1db640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558cfcaccfc0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T23:26:43.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3dc1db640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558cfcacd200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T23:26:43.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3db9da640 -1 rbd::mirror::PoolWatcher: 0x558cfc6701e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:43.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3db9da640 -1 rbd::mirror::PoolWatcher: 0x558cfc6d4960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:43.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3dc1db640 -1 rbd::mirror::PoolWatcher: 0x558cfc6705a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:43.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3d59ce640 -1 rbd::mirror::PoolWatcher: 0x558cfcb09e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:43.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3d59ce640 -1 rbd::mirror::PoolWatcher: 0x558cfadb7e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:43.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.460+0000 7fb3d59ce640 -1 rbd::mirror::PoolWatcher: 0x558cfc4df0e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:43.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.461+0000 7fb3e21e7640 -1 rbd::mirror::LeaderWatcher: 0x558cfbeb2c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-14T23:26:43.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:43.461+0000 7fb3e21e7640 -1 rbd::mirror::LeaderWatcher: 0x558cfbeb2c00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-14T23:26:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:pool 'mirror_parent' does not exist 2024-09-14T23:26:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-14T23:26:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirrors cluster1 2024-09-14T23:26:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1 2024-09-14T23:26:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 0 3 2024-09-14T23:26:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirror cluster1:0 '' 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1:0 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirror cluster1:1 '' 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1:1 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:26:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirror cluster1:2 '' 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1:2 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirror cluster1:3 '' 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster1:3 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T23:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T23:26:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-14T23:26:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirrors cluster2 2024-09-14T23:26:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2 2024-09-14T23:26:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:++ seq 0 3 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirror cluster2:0 '' 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2:0 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirror cluster2:1 '' 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2:1 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T23:26:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirror cluster2:2 '' 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2:2 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ stop_mirror cluster2:3 '' 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local cluster=cluster2:3 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ local sig= 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test -n 1 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ return 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' -z 1 ']' 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ test '' = /tmp/tmp.OxLGv2e3xU 2024-09-14T23:26:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ rm -Rf /tmp/tmp.OxLGv2e3xU 2024-09-14T23:26:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T23:26:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ echo OK 2024-09-14T23:26:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stdout:OK 2024-09-14T23:26:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi089.stderr:+ exit 0 2024-09-14T23:26:43.641 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-14T23:26:43.641 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-14T23:26:43.836 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-14T23:26:43.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-14T23:26:44.108 DEBUG:teuthology.parallel:result is None 2024-09-14T23:26:44.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-14T23:26:44.133 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-14T23:26:44.133 DEBUG:teuthology.orchestra.run.smithi089:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T23:26:44.199 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-14T23:26:44.199 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-14T23:26:44.215 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-14T23:26:44.215 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 20 2024-09-14T23:26:44.215 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-14T23:26:44.215 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-14T23:26:44.229 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-14T23:26:44.229 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 19 2024-09-14T23:26:44.229 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-14T23:26:44.229 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T23:26:44.239 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-14T23:26:44.239 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:26:44.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.239+0000 7f9ea5b6f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 68766) UID: 1000 2024-09-14T23:26:44.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.239+0000 7f9ea536e640 -1 rbd::mirror::LeaderWatcher: 0x558159715200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9ea4b6d640 -1 rbd::mirror::InstanceWatcher: 0x55815952e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6308 object: (2) No such file or directory 2024-09-14T23:26:44.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9ea4b6d640 -1 rbd::mirror::InstanceWatcher: 0x558159a0a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6308 object: (2) No such file or directory 2024-09-14T23:26:44.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9ea4b6d640 -1 rbd::mirror::InstanceWatcher: 0x55815952fc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6311 object: (2) No such file or directory 2024-09-14T23:26:44.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9e9e360640 -1 rbd::mirror::InstanceWatcher: 0x55815952e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9e9e360640 -1 rbd::mirror::InstanceWatcher: 0x558159a0a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9e90b45640 -1 rbd::mirror::InstanceWatcher: 0x55815952fc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9e97b53640 -1 rbd::mirror::MirrorStatusUpdater 0x55815957a5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9ea4b6d640 -1 rbd::mirror::NamespaceReplayer: 0x558159578000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9e8a338640 -1 rbd::mirror::MirrorStatusUpdater 0x55815957a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9ea4b6d640 -1 rbd::mirror::NamespaceReplayer: 0x5581595784e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9e98354640 -1 rbd::mirror::MirrorStatusUpdater 0x55815957a360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.240+0000 7f9ea4b6d640 -1 rbd::mirror::NamespaceReplayer: 0x5581595781a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.241+0000 7f9e9eb61640 -1 rbd::mirror::MirrorStatusUpdater 0x55815957a000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.241+0000 7f9ea536e640 -1 rbd::mirror::NamespaceReplayer: 0x558159578000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.241+0000 7f9e91346640 -1 rbd::mirror::MirrorStatusUpdater 0x558159791200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.241+0000 7f9ea4b6d640 -1 rbd::mirror::NamespaceReplayer: 0x5581595784e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.241+0000 7f9e9e360640 -1 rbd::mirror::MirrorStatusUpdater 0x55815957a120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.241+0000 7f9ea536e640 -1 rbd::mirror::NamespaceReplayer: 0x5581595781a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.243+0000 7f9ea536e640 -1 rbd::mirror::LeaderWatcher: 0x5581594a4c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.243+0000 7f9ea536e640 -1 rbd::mirror::InstanceWatcher: 0x55815952e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6308 object: (2) No such file or directory 2024-09-14T23:26:44.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.243+0000 7f9e9eb61640 -1 rbd::mirror::InstanceWatcher: 0x55815952e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.244+0000 7f9e97b53640 -1 rbd::mirror::MirrorStatusUpdater 0x5581592b3200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.244+0000 7f9ea536e640 -1 rbd::mirror::NamespaceReplayer: 0x5581591761a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.244+0000 7f9e9e360640 -1 rbd::mirror::MirrorStatusUpdater 0x5581592b37a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.244+0000 7f9ea4b6d640 -1 rbd::mirror::NamespaceReplayer: 0x5581591761a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-14T23:26:44.255 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T23:26:44.265 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-14T23:26:44.265 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:26:44.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.296+0000 7fb3e31e9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 51294) UID: 1000 2024-09-14T23:26:44.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.296+0000 7fb3ce1bf640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558cfc6cd440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T23:26:44.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3e29e8640 -1 rbd::mirror::InstanceWatcher: 0x558cfbf3c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5264 object: (2) No such file or directory 2024-09-14T23:26:44.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3e29e8640 -1 rbd::mirror::InstanceWatcher: 0x558cfc3ee380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5264 object: (2) No such file or directory 2024-09-14T23:26:44.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3ce1bf640 -1 rbd::mirror::PoolWatcher: 0x558cfc4df2c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:44.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3dc1db640 -1 rbd::mirror::InstanceWatcher: 0x558cfbf3c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3dc1db640 -1 rbd::mirror::InstanceWatcher: 0x558cfc3ee380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3c79b2640 -1 rbd::mirror::PoolWatcher: 0x558cfc4deb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3d51cd640 -1 rbd::mirror::MirrorStatusUpdater 0x558cfbf88360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3e29e8640 -1 rbd::mirror::NamespaceReplayer: 0x558cfbf86000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3d59ce640 -1 rbd::mirror::MirrorStatusUpdater 0x558cfbf88480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3e29e8640 -1 rbd::mirror::NamespaceReplayer: 0x558cfbf861a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3db9da640 -1 rbd::mirror::MirrorStatusUpdater 0x558cfbf88000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3db9da640 -1 rbd::mirror::MirrorStatusUpdater 0x558cfbf88120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3e29e8640 -1 rbd::mirror::NamespaceReplayer: 0x558cfbf86000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.297+0000 7fb3e29e8640 -1 rbd::mirror::NamespaceReplayer: 0x558cfbf861a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.298+0000 7fb3e21e7640 -1 rbd::mirror::LeaderWatcher: 0x558cfc13b200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.298+0000 7fb3e21e7640 -1 rbd::mirror::LeaderWatcher: 0x558cfc13b200 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.298+0000 7fb3e29e8640 -1 rbd::mirror::LeaderWatcher: 0x558cfc13b200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.298+0000 7fb3e29e8640 -1 rbd::mirror::InstanceWatcher: 0x558cfc3ee000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5267 object: (2) No such file or directory 2024-09-14T23:26:44.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.298+0000 7fb3ce1bf640 -1 rbd::mirror::InstanceWatcher: 0x558cfc3ee000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.299+0000 7fb3c81b3640 -1 rbd::mirror::MirrorStatusUpdater 0x558cfbf886c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.299+0000 7fb3e29e8640 -1 rbd::mirror::NamespaceReplayer: 0x558cfbf864e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.299+0000 7fb3ce9c0640 -1 rbd::mirror::MirrorStatusUpdater 0x558cfc18afc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.299+0000 7fb3e21e7640 -1 rbd::mirror::NamespaceReplayer: 0x558cfbf864e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.301+0000 7fb3e29e8640 -1 rbd::mirror::LeaderWatcher: 0x558cfbeb2c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.301+0000 7fb3e21e7640 -1 rbd::mirror::InstanceWatcher: 0x558cfbf3c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5264 object: (2) No such file or directory 2024-09-14T23:26:44.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.301+0000 7fb3dc1db640 -1 rbd::mirror::InstanceWatcher: 0x558cfbf3c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.302+0000 7fb3d51cd640 -1 rbd::mirror::MirrorStatusUpdater 0x558cfbcf66c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.302+0000 7fb3e21e7640 -1 rbd::mirror::NamespaceReplayer: 0x558cfbbac1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.302+0000 7fb3db9da640 -1 rbd::mirror::MirrorStatusUpdater 0x558cfbca8fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.302+0000 7fb3e29e8640 -1 rbd::mirror::NamespaceReplayer: 0x558cfbbac1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-14T23:26:44.315 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T23:26:44.324 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-14T23:26:44.324 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:26:44.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.355+0000 7f55ff2e5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 70983) UID: 1000 2024-09-14T23:26:44.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.355+0000 7f55fe2e3640 -1 rbd::mirror::LeaderWatcher: 0x55a1d4b3d200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.355+0000 7f55feae4640 -1 rbd::mirror::InstanceWatcher: 0x55a1d4920380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6476 object: (2) No such file or directory 2024-09-14T23:26:44.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.355+0000 7f55feae4640 -1 rbd::mirror::InstanceWatcher: 0x55a1d4920700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6476 object: (2) No such file or directory 2024-09-14T23:26:44.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55feae4640 -1 rbd::mirror::InstanceWatcher: 0x55a1d4dec000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6479 object: (2) No such file or directory 2024-09-14T23:26:44.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55f82d7640 -1 rbd::mirror::InstanceWatcher: 0x55a1d4920380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55f82d7640 -1 rbd::mirror::InstanceWatcher: 0x55a1d4920700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55ea2bb640 -1 rbd::mirror::InstanceWatcher: 0x55a1d4dec000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55f1aca640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d49687e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55fe2e3640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d490c4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55f1aca640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d49685a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55fe2e3640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d490c680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55e3aae640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d4bac6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55feae4640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d490c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.356+0000 7f55f82d7640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d4968000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.357+0000 7f55f82d7640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d4968120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.357+0000 7f55fe2e3640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d490c4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.357+0000 7f55fe2e3640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d490c680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.357+0000 7f55eaabc640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d4b77320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.357+0000 7f55feae4640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d490c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.358+0000 7f55feae4640 -1 rbd::mirror::LeaderWatcher: 0x55a1d487ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.359+0000 7f55feae4640 -1 rbd::mirror::InstanceWatcher: 0x55a1d4920000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6476 object: (2) No such file or directory 2024-09-14T23:26:44.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.359+0000 7f55f7ad6640 -1 rbd::mirror::InstanceWatcher: 0x55a1d4920000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.359+0000 7f55f12c9640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d46cc5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.359+0000 7f55feae4640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d490c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.360+0000 7f55f82d7640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d46850e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi089.stderr:2024-09-14T23:26:44.360+0000 7f55fe2e3640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d490c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-14T23:26:44.371 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T23:26:44.383 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-14T23:26:44.383 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:26:44.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.412+0000 7fa79187d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 74768) UID: 1000 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.412+0000 7fa79087b640 -1 rbd::mirror::LeaderWatcher: 0x55fddb257200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa79087b640 -1 rbd::mirror::InstanceWatcher: 0x55fddb022700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6849 object: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa79087b640 -1 rbd::mirror::InstanceWatcher: 0x55fddb022380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6849 object: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa79107c640 -1 rbd::mirror::InstanceWatcher: 0x55fddb022a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6851 object: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa78a06e640 -1 rbd::mirror::InstanceWatcher: 0x55fddb022700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa78a06e640 -1 rbd::mirror::InstanceWatcher: 0x55fddb022380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa77c853640 -1 rbd::mirror::InstanceWatcher: 0x55fddb022a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa784062640 -1 rbd::mirror::MirrorStatusUpdater 0x55fddb090900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa784062640 -1 rbd::mirror::MirrorStatusUpdater 0x55fddb090360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa79087b640 -1 rbd::mirror::NamespaceReplayer: 0x55fddb08e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa79087b640 -1 rbd::mirror::NamespaceReplayer: 0x55fddb08e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa776847640 -1 rbd::mirror::MirrorStatusUpdater 0x55fddb28c7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa79107c640 -1 rbd::mirror::NamespaceReplayer: 0x55fdd9e36680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa78a06e640 -1 rbd::mirror::MirrorStatusUpdater 0x55fddb090000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa78a06e640 -1 rbd::mirror::MirrorStatusUpdater 0x55fddb090120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa79087b640 -1 rbd::mirror::NamespaceReplayer: 0x55fddb08e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.413+0000 7fa79087b640 -1 rbd::mirror::NamespaceReplayer: 0x55fddb08e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.414+0000 7fa77d054640 -1 rbd::mirror::MirrorStatusUpdater 0x55fddb245200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.414+0000 7fa79107c640 -1 rbd::mirror::NamespaceReplayer: 0x55fdd9e36680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.415+0000 7fa79107c640 -1 rbd::mirror::LeaderWatcher: 0x55fddaf7ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.416+0000 7fa79107c640 -1 rbd::mirror::InstanceWatcher: 0x55fddb022000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6849 object: (2) No such file or directory 2024-09-14T23:26:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.416+0000 7fa78a86f640 -1 rbd::mirror::InstanceWatcher: 0x55fddb022000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.416+0000 7fa783861640 -1 rbd::mirror::MirrorStatusUpdater 0x55fddafa2480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.416+0000 7fa79107c640 -1 rbd::mirror::NamespaceReplayer: 0x55fddb886000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.416+0000 7fa78a06e640 -1 rbd::mirror::MirrorStatusUpdater 0x55fddaddf320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.416+0000 7fa79087b640 -1 rbd::mirror::NamespaceReplayer: 0x55fddb886000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-14T23:26:44.428 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T23:26:44.438 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-14T23:26:44.438 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:26:44.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f578cd86640 -1 rbd::mirror::LeaderWatcher: 0x55e49a48ec00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-14T23:26:44.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f5786d7a640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e49b033200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T23:26:44.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f5786579640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e49b07ab40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T23:26:44.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f5786579640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e49b07a900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T23:26:44.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f76224d9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 75173) UID: 1000 2024-09-14T23:26:44.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f5786d7a640 -1 rbd::mirror::PoolWatcher: 0x55e49a27ba40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:44.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f5786579640 -1 rbd::mirror::PoolWatcher: 0x55e499355c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:44.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f5786579640 -1 rbd::mirror::PoolWatcher: 0x55e49a27b860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:44.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f578056d640 -1 rbd::mirror::PoolWatcher: 0x55e49aad41e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:44.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f578056d640 -1 rbd::mirror::PoolWatcher: 0x55e49aa170e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:44.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f578056d640 -1 rbd::mirror::PoolWatcher: 0x55e49b1205a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:44.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f76214d7640 -1 rbd::mirror::LeaderWatcher: 0x557adb8c7200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f7621cd8640 -1 rbd::mirror::InstanceWatcher: 0x557adb6bc700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6210 object: (2) No such file or directory 2024-09-14T23:26:44.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f578d587640 -1 rbd::mirror::LeaderWatcher: 0x55e49a48ec00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-14T23:26:44.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f7621cd8640 -1 rbd::mirror::InstanceWatcher: 0x557adb6bc380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6210 object: (2) No such file or directory 2024-09-14T23:26:44.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f578d587640 -1 rbd::mirror::LeaderWatcher: 0x55e49a48ec00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-14T23:26:44.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f7621cd8640 -1 rbd::mirror::InstanceWatcher: 0x557adbb9c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6213 object: (2) No such file or directory 2024-09-14T23:26:44.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f761acca640 -1 rbd::mirror::InstanceWatcher: 0x557adb6bc700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.455+0000 7f761acca640 -1 rbd::mirror::InstanceWatcher: 0x557adb6bc380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f7614cbe640 -1 rbd::mirror::MirrorStatusUpdater 0x557adb7387e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f7614cbe640 -1 rbd::mirror::MirrorStatusUpdater 0x557adb738360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f7621cd8640 -1 rbd::mirror::NamespaceReplayer: 0x557adb730000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f760dcb0640 -1 rbd::mirror::InstanceWatcher: 0x557adbb9c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f76214d7640 -1 rbd::mirror::NamespaceReplayer: 0x557adb7301a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f761b4cb640 -1 rbd::mirror::MirrorStatusUpdater 0x557adb738000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f761b4cb640 -1 rbd::mirror::MirrorStatusUpdater 0x557adb738120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f76214d7640 -1 rbd::mirror::NamespaceReplayer: 0x557adb730000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f76214d7640 -1 rbd::mirror::NamespaceReplayer: 0x557adb7301a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f76074a3640 -1 rbd::mirror::MirrorStatusUpdater 0x557adb738900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.456+0000 7f7621cd8640 -1 rbd::mirror::NamespaceReplayer: 0x557adb7304e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.457+0000 7f760d4af640 -1 rbd::mirror::MirrorStatusUpdater 0x557adb8feea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.457+0000 7f76214d7640 -1 rbd::mirror::NamespaceReplayer: 0x557adb7304e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.458+0000 7f76214d7640 -1 rbd::mirror::LeaderWatcher: 0x557adb62cc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.459+0000 7f7621cd8640 -1 rbd::mirror::InstanceWatcher: 0x557adb6bc000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6210 object: (2) No such file or directory 2024-09-14T23:26:44.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.459+0000 7f761acca640 -1 rbd::mirror::InstanceWatcher: 0x557adb6bc000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.459+0000 7f76144bd640 -1 rbd::mirror::MirrorStatusUpdater 0x557adb4786c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.459+0000 7f76214d7640 -1 rbd::mirror::NamespaceReplayer: 0x557ada50c680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.460+0000 7f761b4cb640 -1 rbd::mirror::MirrorStatusUpdater 0x557adb415200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi089.stderr:2024-09-14T23:26:44.460+0000 7f7621cd8640 -1 rbd::mirror::NamespaceReplayer: 0x557ada50c680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-14T23:26:44.471 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T23:26:44.481 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-14T23:26:44.481 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:26:44.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.511+0000 7ff403118640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 70361) UID: 1000 2024-09-14T23:26:44.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.512+0000 7ff402917640 -1 rbd::mirror::InstanceWatcher: 0x56360a1a4380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6034 object: (2) No such file or directory 2024-09-14T23:26:44.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.512+0000 7ff402917640 -1 rbd::mirror::InstanceWatcher: 0x56360a666000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6034 object: (2) No such file or directory 2024-09-14T23:26:44.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.513+0000 7ff3fc10a640 -1 rbd::mirror::InstanceWatcher: 0x56360a1a4380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.513+0000 7ff3fc10a640 -1 rbd::mirror::InstanceWatcher: 0x56360a666000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.514+0000 7ff3f58fd640 -1 rbd::mirror::MirrorStatusUpdater 0x56360a1f0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.514+0000 7ff3f58fd640 -1 rbd::mirror::MirrorStatusUpdater 0x56360a3cca20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.514+0000 7ff402917640 -1 rbd::mirror::NamespaceReplayer: 0x56360a1ee000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.514+0000 7ff402917640 -1 rbd::mirror::NamespaceReplayer: 0x56360a1ee1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.514+0000 7ff3fb909640 -1 rbd::mirror::MirrorStatusUpdater 0x56360a1f0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.514+0000 7ff3fb909640 -1 rbd::mirror::MirrorStatusUpdater 0x56360a1f0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.514+0000 7ff402116640 -1 rbd::mirror::NamespaceReplayer: 0x56360a1ee000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.514+0000 7ff402116640 -1 rbd::mirror::NamespaceReplayer: 0x56360a1ee1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.717+0000 7ff402917640 -1 rbd::mirror::LeaderWatcher: 0x56360a38f200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.717+0000 7ff3ee8ef640 -1 rbd::mirror::InstanceWatcher: 0x56360a1a5c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.717+0000 7ff3e78e1640 -1 rbd::mirror::MirrorStatusUpdater 0x56360a1f06c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.717+0000 7ff402917640 -1 rbd::mirror::NamespaceReplayer: 0x56360a1ee4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.718+0000 7ff3ee0ee640 -1 rbd::mirror::MirrorStatusUpdater 0x56360a3b8fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.718+0000 7ff402116640 -1 rbd::mirror::NamespaceReplayer: 0x56360a1ee4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.720+0000 7ff402116640 -1 rbd::mirror::LeaderWatcher: 0x56360a10ac00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:44.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.720+0000 7ff402116640 -1 rbd::mirror::InstanceWatcher: 0x56360a1a4000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6034 object: (2) No such file or directory 2024-09-14T23:26:44.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.721+0000 7ff3fc10a640 -1 rbd::mirror::InstanceWatcher: 0x56360a1a4000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:44.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.721+0000 7ff3f50fc640 -1 rbd::mirror::MirrorStatusUpdater 0x563609f5c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.721+0000 7ff402116640 -1 rbd::mirror::NamespaceReplayer: 0x563608fc81a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:44.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.721+0000 7ff3fb909640 -1 rbd::mirror::MirrorStatusUpdater 0x563609f010e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:44.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi089.stderr:2024-09-14T23:26:44.721+0000 7ff402917640 -1 rbd::mirror::NamespaceReplayer: 0x563608fc81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:45.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:45.382+0000 7f578cd86640 -1 rbd::mirror::LeaderWatcher: 0x55e49a743200 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-14T23:26:45.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:45.383+0000 7f5778d5e640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e49a480480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T23:26:45.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:45.383+0000 7f577955f640 -1 rbd::mirror::PoolWatcher: 0x55e49aa17680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T23:26:45.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:45.384+0000 7f578cd86640 -1 rbd::mirror::LeaderWatcher: 0x55e49a743200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-14T23:26:45.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:45.384+0000 7f578cd86640 -1 rbd::mirror::LeaderWatcher: 0x55e49a743200 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-14T23:26:47.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.522+0000 7f1cb8af4640 -1 rbd::mirror::RemotePoolPoller: 0x565465392ea0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-14T23:26:47.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.522+0000 7f1cb82f3640 -1 rbd::mirror::RemotePoolPoller: 0x565465392ea0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-14T23:26:47.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.522+0000 7f1cb82f3640 -1 rbd::mirror::PoolReplayer: 0x5654653a5b00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-14T23:26:47.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.523+0000 7f1cc530d640 -1 rbd::mirror::InstanceWatcher: 0x565466556380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6171 object: (2) No such file or directory 2024-09-14T23:26:47.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.523+0000 7f1cc530d640 -1 rbd::mirror::InstanceWatcher: 0x565466a2c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6171 object: (2) No such file or directory 2024-09-14T23:26:47.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cbeb00640 -1 rbd::mirror::InstanceWatcher: 0x565466556380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:47.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cbeb00640 -1 rbd::mirror::InstanceWatcher: 0x565466a2c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:47.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cb8af4640 -1 rbd::mirror::MirrorStatusUpdater 0x5654665a4360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:47.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cb8af4640 -1 rbd::mirror::MirrorStatusUpdater 0x56546679aa20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:47.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cc530d640 -1 rbd::mirror::NamespaceReplayer: 0x5654665a2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:47.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cc530d640 -1 rbd::mirror::NamespaceReplayer: 0x5654665a21a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:47.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cbf301640 -1 rbd::mirror::MirrorStatusUpdater 0x5654665a4000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:47.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cbf301640 -1 rbd::mirror::MirrorStatusUpdater 0x5654665a4120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:47.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cc5b0e640 -1 rbd::mirror::NamespaceReplayer: 0x5654665a2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:47.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.524+0000 7f1cc5b0e640 -1 rbd::mirror::NamespaceReplayer: 0x5654665a21a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:47.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.577+0000 7f1caaad8640 -1 rbd::mirror::RemotePoolPoller: 0x565465393ad0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-14T23:26:47.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.577+0000 7f1cab2d9640 -1 rbd::mirror::RemotePoolPoller: 0x565465393ad0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-14T23:26:47.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.577+0000 7f1cab2d9640 -1 rbd::mirror::PoolReplayer: 0x5654664bcf00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-14T23:26:47.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.618+0000 7f1cc5b0e640 -1 rbd::mirror::LeaderWatcher: 0x5654664bcc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:47.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.619+0000 7f1cc5b0e640 -1 rbd::mirror::InstanceWatcher: 0x565466556000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6171 object: (2) No such file or directory 2024-09-14T23:26:47.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.619+0000 7f1cbeb00640 -1 rbd::mirror::InstanceWatcher: 0x565466556000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:47.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.619+0000 7f1cb82f3640 -1 rbd::mirror::MirrorStatusUpdater 0x56546630e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:47.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.619+0000 7f1cc5b0e640 -1 rbd::mirror::NamespaceReplayer: 0x565466db6000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:47.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.620+0000 7f1cbf301640 -1 rbd::mirror::MirrorStatusUpdater 0x5654662a70e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:47.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.620+0000 7f1cc530d640 -1 rbd::mirror::NamespaceReplayer: 0x565466db6000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:47.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.623+0000 7f1cb12e5640 -1 rbd::mirror::InstanceWatcher: 0x565466556700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:47.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.623+0000 7f1caaad8640 -1 rbd::mirror::MirrorStatusUpdater 0x5654665a4480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:47.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.623+0000 7f1cc530d640 -1 rbd::mirror::NamespaceReplayer: 0x5654665a24e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:47.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.624+0000 7f1cb1ae6640 -1 rbd::mirror::MirrorStatusUpdater 0x565466766fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:47.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:47.624+0000 7f1cc5b0e640 -1 rbd::mirror::NamespaceReplayer: 0x5654665a24e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:50.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-14T23:26:50.583 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T23:26:50.595 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-14T23:26:50.596 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:26:50.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi089.stderr:2024-09-14T23:26:50.595+0000 7f1cc630f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 74041) UID: 1000 2024-09-14T23:26:50.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-14T23:26:50.608 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T23:26:50.619 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-14T23:26:50.619 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:26:50.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.649+0000 7f578dd88640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 65922) UID: 1000 2024-09-14T23:26:50.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.649+0000 7f578d587640 -1 rbd::mirror::LeaderWatcher: 0x55e49a743200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:50.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.649+0000 7f578d587640 -1 rbd::mirror::InstanceWatcher: 0x55e49a522380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5726 object: (2) No such file or directory 2024-09-14T23:26:50.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.649+0000 7f578d587640 -1 rbd::mirror::InstanceWatcher: 0x55e49aa0e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5726 object: (2) No such file or directory 2024-09-14T23:26:50.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.649+0000 7f578cd86640 -1 rbd::mirror::InstanceWatcher: 0x55e49a523c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5729 object: (2) No such file or directory 2024-09-14T23:26:50.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.649+0000 7f5786579640 -1 rbd::mirror::InstanceWatcher: 0x55e49a522380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:50.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f5786d7a640 -1 rbd::mirror::InstanceWatcher: 0x55e49aa0e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f5778d5e640 -1 rbd::mirror::InstanceWatcher: 0x55e49a523c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f577fd6c640 -1 rbd::mirror::MirrorStatusUpdater 0x55e49a598360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f577fd6c640 -1 rbd::mirror::MirrorStatusUpdater 0x55e49a598480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f578d587640 -1 rbd::mirror::NamespaceReplayer: 0x55e49a594000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f578d587640 -1 rbd::mirror::NamespaceReplayer: 0x55e49a5941a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f5786579640 -1 rbd::mirror::MirrorStatusUpdater 0x55e49a598000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f5786579640 -1 rbd::mirror::MirrorStatusUpdater 0x55e49a598120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f578cd86640 -1 rbd::mirror::NamespaceReplayer: 0x55e49a594000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f578cd86640 -1 rbd::mirror::NamespaceReplayer: 0x55e49a5941a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:50.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f577955f640 -1 rbd::mirror::MirrorStatusUpdater 0x55e49a76efc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:50.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.650+0000 7f578d587640 -1 rbd::mirror::NamespaceReplayer: 0x55e49a5944e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:50.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.653+0000 7f578cd86640 -1 rbd::mirror::LeaderWatcher: 0x55e49a48ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T23:26:50.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.653+0000 7f578d587640 -1 rbd::mirror::InstanceWatcher: 0x55e49a522000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5726 object: (2) No such file or directory 2024-09-14T23:26:50.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.654+0000 7f5786d7a640 -1 rbd::mirror::InstanceWatcher: 0x55e49a522000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T23:26:50.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.654+0000 7f578056d640 -1 rbd::mirror::MirrorStatusUpdater 0x55e49a2ea6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:50.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.654+0000 7f578d587640 -1 rbd::mirror::NamespaceReplayer: 0x55e49934e680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T23:26:50.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.654+0000 7f5786579640 -1 rbd::mirror::MirrorStatusUpdater 0x55e49a26f0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T23:26:50.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi089.stderr:2024-09-14T23:26:50.654+0000 7f578cd86640 -1 rbd::mirror::NamespaceReplayer: 0x55e49934e680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T23:26:50.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-14T23:26:50.671 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-14T23:26:50.686 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-14T23:26:50.686 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T23:26:51.220 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T23:26:51.220 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-14T23:26:51.231 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":1510,"stamp":"2024-09-14T23:26:49.452216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4149360,"kb_used_data":9456,"kb_used_omap":4,"kb_used_meta":4139835,"kb_avail":278966160,"statfs":{"total":289910292480,"available":285661347840,"internally_reserved":0,"allocated":9682944,"data_stored":6283150,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":4239191377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-381661238,"num_objects":-127,"num_object_clones":0,"num_object_copies":-254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-127,"num_whiteouts":0,"num_read":-118677,"num_read_kb":-39620720,"num_write":-608391,"num_write_kb":-224718066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-10,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002033"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.518885+0000","last_change":"2024-09-14T22:49:29.945463+0000","last_active":"2024-09-14T23:26:43.518885+0000","last_peered":"2024-09-14T23:26:43.518885+0000","last_clean":"2024-09-14T23:26:43.518885+0000","last_became_active":"2024-09-14T22:49:27.947764+0000","last_became_peered":"2024-09-14T22:49:27.947764+0000","last_unstale":"2024-09-14T23:26:43.518885+0000","last_undegraded":"2024-09-14T23:26:43.518885+0000","last_fullsized":"2024-09-14T23:26:43.518885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:23:56.112794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096834600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.519540+0000","last_change":"2024-09-14T22:49:29.945152+0000","last_active":"2024-09-14T23:26:43.519540+0000","last_peered":"2024-09-14T23:26:43.519540+0000","last_clean":"2024-09-14T23:26:43.519540+0000","last_became_active":"2024-09-14T22:49:27.948873+0000","last_became_peered":"2024-09-14T22:49:27.948873+0000","last_unstale":"2024-09-14T23:26:43.519540+0000","last_undegraded":"2024-09-14T23:26:43.519540+0000","last_fullsized":"2024-09-14T23:26:43.519540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:59.551999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069214500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.517481+0000","last_change":"2024-09-14T22:49:29.945105+0000","last_active":"2024-09-14T23:26:43.517481+0000","last_peered":"2024-09-14T23:26:43.517481+0000","last_clean":"2024-09-14T23:26:43.517481+0000","last_became_active":"2024-09-14T22:49:27.949153+0000","last_became_peered":"2024-09-14T22:49:27.949153+0000","last_unstale":"2024-09-14T23:26:43.517481+0000","last_undegraded":"2024-09-14T23:26:43.517481+0000","last_fullsized":"2024-09-14T23:26:43.517481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:58:19.694736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000710178,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.516494+0000","last_change":"2024-09-14T22:49:29.944792+0000","last_active":"2024-09-14T23:26:43.516494+0000","last_peered":"2024-09-14T23:26:43.516494+0000","last_clean":"2024-09-14T23:26:43.516494+0000","last_became_active":"2024-09-14T22:49:27.947823+0000","last_became_peered":"2024-09-14T22:49:27.947823+0000","last_unstale":"2024-09-14T23:26:43.516494+0000","last_undegraded":"2024-09-14T23:26:43.516494+0000","last_fullsized":"2024-09-14T23:26:43.516494+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:16:27.031276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043000900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.759873+0000","last_change":"2024-09-14T22:49:29.949279+0000","last_active":"2024-09-14T23:26:43.759873+0000","last_peered":"2024-09-14T23:26:43.759873+0000","last_clean":"2024-09-14T23:26:43.759873+0000","last_became_active":"2024-09-14T22:49:27.948120+0000","last_became_peered":"2024-09-14T22:49:27.948120+0000","last_unstale":"2024-09-14T23:26:43.759873+0000","last_undegraded":"2024-09-14T23:26:43.759873+0000","last_fullsized":"2024-09-14T23:26:43.759873+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:53:56.777188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095602899999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":777,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.772475+0000","last_change":"2024-09-14T22:49:29.973218+0000","last_active":"2024-09-14T23:26:43.772475+0000","last_peered":"2024-09-14T23:26:43.772475+0000","last_clean":"2024-09-14T23:26:43.772475+0000","last_became_active":"2024-09-14T22:49:27.948225+0000","last_became_peered":"2024-09-14T22:49:27.948225+0000","last_unstale":"2024-09-14T23:26:43.772475+0000","last_undegraded":"2024-09-14T23:26:43.772475+0000","last_fullsized":"2024-09-14T23:26:43.772475+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:34.949309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044858000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.770817+0000","last_change":"2024-09-14T22:49:29.973217+0000","last_active":"2024-09-14T23:26:43.770817+0000","last_peered":"2024-09-14T23:26:43.770817+0000","last_clean":"2024-09-14T23:26:43.770817+0000","last_became_active":"2024-09-14T22:49:27.947929+0000","last_became_peered":"2024-09-14T22:49:27.947929+0000","last_unstale":"2024-09-14T23:26:43.770817+0000","last_undegraded":"2024-09-14T23:26:43.770817+0000","last_fullsized":"2024-09-14T23:26:43.770817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:21:03.176471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.518506+0000","last_change":"2024-09-14T22:49:29.944800+0000","last_active":"2024-09-14T23:26:43.518506+0000","last_peered":"2024-09-14T23:26:43.518506+0000","last_clean":"2024-09-14T23:26:43.518506+0000","last_became_active":"2024-09-14T22:49:27.947559+0000","last_became_peered":"2024-09-14T22:49:27.947559+0000","last_unstale":"2024-09-14T23:26:43.518506+0000","last_undegraded":"2024-09-14T23:26:43.518506+0000","last_fullsized":"2024-09-14T23:26:43.518506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:33:43.576216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044691000000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":907,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.519178+0000","last_change":"2024-09-14T22:49:22.921000+0000","last_active":"2024-09-14T23:26:43.519178+0000","last_peered":"2024-09-14T23:26:43.519178+0000","last_clean":"2024-09-14T23:26:43.519178+0000","last_became_active":"2024-09-14T22:49:22.919993+0000","last_became_peered":"2024-09-14T22:49:22.919993+0000","last_unstale":"2024-09-14T23:26:43.519178+0000","last_undegraded":"2024-09-14T23:26:43.519178+0000","last_fullsized":"2024-09-14T23:26:43.519178+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_clean_scrub_stamp":"2024-09-14T22:49:21.912115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:51.033680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706118,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1248508,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":1245374,"kb_avail":93123332,"statfs":{"total":96636764160,"available":95358291968,"internally_reserved":0,"allocated":3186688,"data_stored":1700792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1275263429},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706118,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1588088,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1584894,"kb_avail":92783752,"statfs":{"total":96636764160,"available":95010562048,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1622931910},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1312764,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1309566,"kb_avail":93059076,"statfs":{"total":96636764160,"available":95292493824,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1340996038},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:26:51.232 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T23:26:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T23:26:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-14T23:26:51.519 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":1511,"stamp":"2024-09-14T23:26:51.452709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4149360,"kb_used_data":9456,"kb_used_omap":4,"kb_used_meta":4139835,"kb_avail":278966160,"statfs":{"total":289910292480,"available":285661347840,"internally_reserved":0,"allocated":9682944,"data_stored":6283150,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":4239191377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2180,"num_read_kb":-2055,"num_write":-429,"num_write_kb":-319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.943341"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.518885+0000","last_change":"2024-09-14T22:49:29.945463+0000","last_active":"2024-09-14T23:26:43.518885+0000","last_peered":"2024-09-14T23:26:43.518885+0000","last_clean":"2024-09-14T23:26:43.518885+0000","last_became_active":"2024-09-14T22:49:27.947764+0000","last_became_peered":"2024-09-14T22:49:27.947764+0000","last_unstale":"2024-09-14T23:26:43.518885+0000","last_undegraded":"2024-09-14T23:26:43.518885+0000","last_fullsized":"2024-09-14T23:26:43.518885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:23:56.112794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096834600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.519540+0000","last_change":"2024-09-14T22:49:29.945152+0000","last_active":"2024-09-14T23:26:43.519540+0000","last_peered":"2024-09-14T23:26:43.519540+0000","last_clean":"2024-09-14T23:26:43.519540+0000","last_became_active":"2024-09-14T22:49:27.948873+0000","last_became_peered":"2024-09-14T22:49:27.948873+0000","last_unstale":"2024-09-14T23:26:43.519540+0000","last_undegraded":"2024-09-14T23:26:43.519540+0000","last_fullsized":"2024-09-14T23:26:43.519540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:59.551999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069214500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.517481+0000","last_change":"2024-09-14T22:49:29.945105+0000","last_active":"2024-09-14T23:26:43.517481+0000","last_peered":"2024-09-14T23:26:43.517481+0000","last_clean":"2024-09-14T23:26:43.517481+0000","last_became_active":"2024-09-14T22:49:27.949153+0000","last_became_peered":"2024-09-14T22:49:27.949153+0000","last_unstale":"2024-09-14T23:26:43.517481+0000","last_undegraded":"2024-09-14T23:26:43.517481+0000","last_fullsized":"2024-09-14T23:26:43.517481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:58:19.694736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000710178,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.516494+0000","last_change":"2024-09-14T22:49:29.944792+0000","last_active":"2024-09-14T23:26:43.516494+0000","last_peered":"2024-09-14T23:26:43.516494+0000","last_clean":"2024-09-14T23:26:43.516494+0000","last_became_active":"2024-09-14T22:49:27.947823+0000","last_became_peered":"2024-09-14T22:49:27.947823+0000","last_unstale":"2024-09-14T23:26:43.516494+0000","last_undegraded":"2024-09-14T23:26:43.516494+0000","last_fullsized":"2024-09-14T23:26:43.516494+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:16:27.031276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043000900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.759873+0000","last_change":"2024-09-14T22:49:29.949279+0000","last_active":"2024-09-14T23:26:43.759873+0000","last_peered":"2024-09-14T23:26:43.759873+0000","last_clean":"2024-09-14T23:26:43.759873+0000","last_became_active":"2024-09-14T22:49:27.948120+0000","last_became_peered":"2024-09-14T22:49:27.948120+0000","last_unstale":"2024-09-14T23:26:43.759873+0000","last_undegraded":"2024-09-14T23:26:43.759873+0000","last_fullsized":"2024-09-14T23:26:43.759873+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:53:56.777188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095602899999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":777,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.772475+0000","last_change":"2024-09-14T22:49:29.973218+0000","last_active":"2024-09-14T23:26:43.772475+0000","last_peered":"2024-09-14T23:26:43.772475+0000","last_clean":"2024-09-14T23:26:43.772475+0000","last_became_active":"2024-09-14T22:49:27.948225+0000","last_became_peered":"2024-09-14T22:49:27.948225+0000","last_unstale":"2024-09-14T23:26:43.772475+0000","last_undegraded":"2024-09-14T23:26:43.772475+0000","last_fullsized":"2024-09-14T23:26:43.772475+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:34.949309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044858000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.770817+0000","last_change":"2024-09-14T22:49:29.973217+0000","last_active":"2024-09-14T23:26:43.770817+0000","last_peered":"2024-09-14T23:26:43.770817+0000","last_clean":"2024-09-14T23:26:43.770817+0000","last_became_active":"2024-09-14T22:49:27.947929+0000","last_became_peered":"2024-09-14T22:49:27.947929+0000","last_unstale":"2024-09-14T23:26:43.770817+0000","last_undegraded":"2024-09-14T23:26:43.770817+0000","last_fullsized":"2024-09-14T23:26:43.770817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:21:03.176471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.518506+0000","last_change":"2024-09-14T22:49:29.944800+0000","last_active":"2024-09-14T23:26:43.518506+0000","last_peered":"2024-09-14T23:26:43.518506+0000","last_clean":"2024-09-14T23:26:43.518506+0000","last_became_active":"2024-09-14T22:49:27.947559+0000","last_became_peered":"2024-09-14T22:49:27.947559+0000","last_unstale":"2024-09-14T23:26:43.518506+0000","last_undegraded":"2024-09-14T23:26:43.518506+0000","last_fullsized":"2024-09-14T23:26:43.518506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:33:43.576216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044691000000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":907,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.519178+0000","last_change":"2024-09-14T22:49:22.921000+0000","last_active":"2024-09-14T23:26:43.519178+0000","last_peered":"2024-09-14T23:26:43.519178+0000","last_clean":"2024-09-14T23:26:43.519178+0000","last_became_active":"2024-09-14T22:49:22.919993+0000","last_became_peered":"2024-09-14T22:49:22.919993+0000","last_unstale":"2024-09-14T23:26:43.519178+0000","last_undegraded":"2024-09-14T23:26:43.519178+0000","last_fullsized":"2024-09-14T23:26:43.519178+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_clean_scrub_stamp":"2024-09-14T22:49:21.912115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:51.033680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706118,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1248508,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":1245374,"kb_avail":93123332,"statfs":{"total":96636764160,"available":95358291968,"internally_reserved":0,"allocated":3186688,"data_stored":1700792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1275263429},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706118,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1588088,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1584894,"kb_avail":92783752,"statfs":{"total":96636764160,"available":95010562048,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1622931910},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1312764,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1309566,"kb_avail":93059076,"statfs":{"total":96636764160,"available":95292493824,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1340996038},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:26:51.520 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-14T23:26:51.520 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T23:26:51.792 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T23:26:51.793 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-14T23:26:51.805 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":1511,"stamp":"2024-09-14T23:26:51.452709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4149360,"kb_used_data":9456,"kb_used_omap":4,"kb_used_meta":4139835,"kb_avail":278966160,"statfs":{"total":289910292480,"available":285661347840,"internally_reserved":0,"allocated":9682944,"data_stored":6283150,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":4239191377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2180,"num_read_kb":-2055,"num_write":-429,"num_write_kb":-319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.943341"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.518885+0000","last_change":"2024-09-14T22:49:29.945463+0000","last_active":"2024-09-14T23:26:43.518885+0000","last_peered":"2024-09-14T23:26:43.518885+0000","last_clean":"2024-09-14T23:26:43.518885+0000","last_became_active":"2024-09-14T22:49:27.947764+0000","last_became_peered":"2024-09-14T22:49:27.947764+0000","last_unstale":"2024-09-14T23:26:43.518885+0000","last_undegraded":"2024-09-14T23:26:43.518885+0000","last_fullsized":"2024-09-14T23:26:43.518885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:23:56.112794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096834600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.519540+0000","last_change":"2024-09-14T22:49:29.945152+0000","last_active":"2024-09-14T23:26:43.519540+0000","last_peered":"2024-09-14T23:26:43.519540+0000","last_clean":"2024-09-14T23:26:43.519540+0000","last_became_active":"2024-09-14T22:49:27.948873+0000","last_became_peered":"2024-09-14T22:49:27.948873+0000","last_unstale":"2024-09-14T23:26:43.519540+0000","last_undegraded":"2024-09-14T23:26:43.519540+0000","last_fullsized":"2024-09-14T23:26:43.519540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:59.551999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069214500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.517481+0000","last_change":"2024-09-14T22:49:29.945105+0000","last_active":"2024-09-14T23:26:43.517481+0000","last_peered":"2024-09-14T23:26:43.517481+0000","last_clean":"2024-09-14T23:26:43.517481+0000","last_became_active":"2024-09-14T22:49:27.949153+0000","last_became_peered":"2024-09-14T22:49:27.949153+0000","last_unstale":"2024-09-14T23:26:43.517481+0000","last_undegraded":"2024-09-14T23:26:43.517481+0000","last_fullsized":"2024-09-14T23:26:43.517481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:58:19.694736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000710178,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.516494+0000","last_change":"2024-09-14T22:49:29.944792+0000","last_active":"2024-09-14T23:26:43.516494+0000","last_peered":"2024-09-14T23:26:43.516494+0000","last_clean":"2024-09-14T23:26:43.516494+0000","last_became_active":"2024-09-14T22:49:27.947823+0000","last_became_peered":"2024-09-14T22:49:27.947823+0000","last_unstale":"2024-09-14T23:26:43.516494+0000","last_undegraded":"2024-09-14T23:26:43.516494+0000","last_fullsized":"2024-09-14T23:26:43.516494+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:16:27.031276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043000900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.759873+0000","last_change":"2024-09-14T22:49:29.949279+0000","last_active":"2024-09-14T23:26:43.759873+0000","last_peered":"2024-09-14T23:26:43.759873+0000","last_clean":"2024-09-14T23:26:43.759873+0000","last_became_active":"2024-09-14T22:49:27.948120+0000","last_became_peered":"2024-09-14T22:49:27.948120+0000","last_unstale":"2024-09-14T23:26:43.759873+0000","last_undegraded":"2024-09-14T23:26:43.759873+0000","last_fullsized":"2024-09-14T23:26:43.759873+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:53:56.777188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095602899999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":777,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.772475+0000","last_change":"2024-09-14T22:49:29.973218+0000","last_active":"2024-09-14T23:26:43.772475+0000","last_peered":"2024-09-14T23:26:43.772475+0000","last_clean":"2024-09-14T23:26:43.772475+0000","last_became_active":"2024-09-14T22:49:27.948225+0000","last_became_peered":"2024-09-14T22:49:27.948225+0000","last_unstale":"2024-09-14T23:26:43.772475+0000","last_undegraded":"2024-09-14T23:26:43.772475+0000","last_fullsized":"2024-09-14T23:26:43.772475+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:34.949309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044858000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.770817+0000","last_change":"2024-09-14T22:49:29.973217+0000","last_active":"2024-09-14T23:26:43.770817+0000","last_peered":"2024-09-14T23:26:43.770817+0000","last_clean":"2024-09-14T23:26:43.770817+0000","last_became_active":"2024-09-14T22:49:27.947929+0000","last_became_peered":"2024-09-14T22:49:27.947929+0000","last_unstale":"2024-09-14T23:26:43.770817+0000","last_undegraded":"2024-09-14T23:26:43.770817+0000","last_fullsized":"2024-09-14T23:26:43.770817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:21:03.176471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.518506+0000","last_change":"2024-09-14T22:49:29.944800+0000","last_active":"2024-09-14T23:26:43.518506+0000","last_peered":"2024-09-14T23:26:43.518506+0000","last_clean":"2024-09-14T23:26:43.518506+0000","last_became_active":"2024-09-14T22:49:27.947559+0000","last_became_peered":"2024-09-14T22:49:27.947559+0000","last_unstale":"2024-09-14T23:26:43.518506+0000","last_undegraded":"2024-09-14T23:26:43.518506+0000","last_fullsized":"2024-09-14T23:26:43.518506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:33:43.576216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044691000000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":907,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.519178+0000","last_change":"2024-09-14T22:49:22.921000+0000","last_active":"2024-09-14T23:26:43.519178+0000","last_peered":"2024-09-14T23:26:43.519178+0000","last_clean":"2024-09-14T23:26:43.519178+0000","last_became_active":"2024-09-14T22:49:22.919993+0000","last_became_peered":"2024-09-14T22:49:22.919993+0000","last_unstale":"2024-09-14T23:26:43.519178+0000","last_undegraded":"2024-09-14T23:26:43.519178+0000","last_fullsized":"2024-09-14T23:26:43.519178+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_clean_scrub_stamp":"2024-09-14T22:49:21.912115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:51.033680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706118,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1248508,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":1245374,"kb_avail":93123332,"statfs":{"total":96636764160,"available":95358291968,"internally_reserved":0,"allocated":3186688,"data_stored":1700792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1275263429},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706118,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1588088,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1584894,"kb_avail":92783752,"statfs":{"total":96636764160,"available":95010562048,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1622931910},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1312764,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1309566,"kb_avail":93059076,"statfs":{"total":96636764160,"available":95292493824,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1340996038},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:26:51.806 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T23:26:52.080 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T23:26:52.080 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":387,"fsid":"666df62e-9042-4f73-97d4-b9e02ade1b08","created":"2024-09-14T22:49:11.368946+0000","modified":"2024-09-14T23:26:43.508795+0000","last_up_change":"2024-09-14T22:49:20.911648+0000","last_in_change":"2024-09-14T22:49:12.196753+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T22:49:21.064218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T22:49:26.396017+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9565fd8-ae8a-4ebe-af3b-2c4cac8f4eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6801","nonce":1763187606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6803","nonce":1763187606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6807","nonce":1763187606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1763187606},{"type":"v1","addr":"172.21.15.89:6805","nonce":1763187606}]},"public_addr":"172.21.15.89:6801/1763187606","cluster_addr":"172.21.15.89:6803/1763187606","heartbeat_back_addr":"172.21.15.89:6807/1763187606","heartbeat_front_addr":"172.21.15.89:6805/1763187606","state":["exists","up"]},{"osd":1,"uuid":"604fb917-9fee-4429-82c2-e8f45cc16900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6809","nonce":2899321748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6811","nonce":2899321748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6815","nonce":2899321748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2899321748},{"type":"v1","addr":"172.21.15.89:6813","nonce":2899321748}]},"public_addr":"172.21.15.89:6809/2899321748","cluster_addr":"172.21.15.89:6811/2899321748","heartbeat_back_addr":"172.21.15.89:6815/2899321748","heartbeat_front_addr":"172.21.15.89:6813/2899321748","state":["exists","up"]},{"osd":2,"uuid":"670d215b-6b09-4e95-a9eb-f18c22b78f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6817","nonce":35213865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6819","nonce":35213865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6823","nonce":35213865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":35213865},{"type":"v1","addr":"172.21.15.89:6821","nonce":35213865}]},"public_addr":"172.21.15.89:6817/35213865","cluster_addr":"172.21.15.89:6819/35213865","heartbeat_back_addr":"172.21.15.89:6823/35213865","heartbeat_front_addr":"172.21.15.89:6821/35213865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:18.864288+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:18.455058+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:49:19.337726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T23:26:53.093 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-14T23:26:53.093 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-14T23:26:53.227 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-14T23:26:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T23:26:53.227 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-14T23:26:53.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-14T23:26:53.514 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 0 to deep-scrub 2024-09-14T23:26:53.527 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-14T23:26:53.527 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-14T23:26:53.648 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-14T23:26:53.648 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T23:26:53.648 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-14T23:26:53.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-14T23:26:53.932 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 1 to deep-scrub 2024-09-14T23:26:53.946 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-14T23:26:53.946 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-14T23:26:54.067 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-14T23:26:54.067 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T23:26:54.067 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-14T23:26:54.078 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-14T23:26:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 2 to deep-scrub 2024-09-14T23:26:54.365 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T23:26:54.637 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T23:26:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-14T23:26:54.650 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":1512,"stamp":"2024-09-14T23:26:53.453091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4149360,"kb_used_data":9456,"kb_used_omap":4,"kb_used_meta":4139835,"kb_avail":278966160,"statfs":{"total":289910292480,"available":285661347840,"internally_reserved":0,"allocated":9682944,"data_stored":6283150,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":4239191377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2180,"num_read_kb":-2055,"num_write":-429,"num_write_kb":-319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002335"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.518885+0000","last_change":"2024-09-14T22:49:29.945463+0000","last_active":"2024-09-14T23:26:43.518885+0000","last_peered":"2024-09-14T23:26:43.518885+0000","last_clean":"2024-09-14T23:26:43.518885+0000","last_became_active":"2024-09-14T22:49:27.947764+0000","last_became_peered":"2024-09-14T22:49:27.947764+0000","last_unstale":"2024-09-14T23:26:43.518885+0000","last_undegraded":"2024-09-14T23:26:43.518885+0000","last_fullsized":"2024-09-14T23:26:43.518885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:23:56.112794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096834600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.519540+0000","last_change":"2024-09-14T22:49:29.945152+0000","last_active":"2024-09-14T23:26:43.519540+0000","last_peered":"2024-09-14T23:26:43.519540+0000","last_clean":"2024-09-14T23:26:43.519540+0000","last_became_active":"2024-09-14T22:49:27.948873+0000","last_became_peered":"2024-09-14T22:49:27.948873+0000","last_unstale":"2024-09-14T23:26:43.519540+0000","last_undegraded":"2024-09-14T23:26:43.519540+0000","last_fullsized":"2024-09-14T23:26:43.519540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:59.551999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069214500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.517481+0000","last_change":"2024-09-14T22:49:29.945105+0000","last_active":"2024-09-14T23:26:43.517481+0000","last_peered":"2024-09-14T23:26:43.517481+0000","last_clean":"2024-09-14T23:26:43.517481+0000","last_became_active":"2024-09-14T22:49:27.949153+0000","last_became_peered":"2024-09-14T22:49:27.949153+0000","last_unstale":"2024-09-14T23:26:43.517481+0000","last_undegraded":"2024-09-14T23:26:43.517481+0000","last_fullsized":"2024-09-14T23:26:43.517481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:58:19.694736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000710178,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.516494+0000","last_change":"2024-09-14T22:49:29.944792+0000","last_active":"2024-09-14T23:26:43.516494+0000","last_peered":"2024-09-14T23:26:43.516494+0000","last_clean":"2024-09-14T23:26:43.516494+0000","last_became_active":"2024-09-14T22:49:27.947823+0000","last_became_peered":"2024-09-14T22:49:27.947823+0000","last_unstale":"2024-09-14T23:26:43.516494+0000","last_undegraded":"2024-09-14T23:26:43.516494+0000","last_fullsized":"2024-09-14T23:26:43.516494+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:16:27.031276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043000900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.759873+0000","last_change":"2024-09-14T22:49:29.949279+0000","last_active":"2024-09-14T23:26:43.759873+0000","last_peered":"2024-09-14T23:26:43.759873+0000","last_clean":"2024-09-14T23:26:43.759873+0000","last_became_active":"2024-09-14T22:49:27.948120+0000","last_became_peered":"2024-09-14T22:49:27.948120+0000","last_unstale":"2024-09-14T23:26:43.759873+0000","last_undegraded":"2024-09-14T23:26:43.759873+0000","last_fullsized":"2024-09-14T23:26:43.759873+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:53:56.777188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095602899999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":777,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.772475+0000","last_change":"2024-09-14T22:49:29.973218+0000","last_active":"2024-09-14T23:26:43.772475+0000","last_peered":"2024-09-14T23:26:43.772475+0000","last_clean":"2024-09-14T23:26:43.772475+0000","last_became_active":"2024-09-14T22:49:27.948225+0000","last_became_peered":"2024-09-14T22:49:27.948225+0000","last_unstale":"2024-09-14T23:26:43.772475+0000","last_undegraded":"2024-09-14T23:26:43.772475+0000","last_fullsized":"2024-09-14T23:26:43.772475+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:34.949309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044858000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.770817+0000","last_change":"2024-09-14T22:49:29.973217+0000","last_active":"2024-09-14T23:26:43.770817+0000","last_peered":"2024-09-14T23:26:43.770817+0000","last_clean":"2024-09-14T23:26:43.770817+0000","last_became_active":"2024-09-14T22:49:27.947929+0000","last_became_peered":"2024-09-14T22:49:27.947929+0000","last_unstale":"2024-09-14T23:26:43.770817+0000","last_undegraded":"2024-09-14T23:26:43.770817+0000","last_fullsized":"2024-09-14T23:26:43.770817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:21:03.176471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.518506+0000","last_change":"2024-09-14T22:49:29.944800+0000","last_active":"2024-09-14T23:26:43.518506+0000","last_peered":"2024-09-14T23:26:43.518506+0000","last_clean":"2024-09-14T23:26:43.518506+0000","last_became_active":"2024-09-14T22:49:27.947559+0000","last_became_peered":"2024-09-14T22:49:27.947559+0000","last_unstale":"2024-09-14T23:26:43.518506+0000","last_undegraded":"2024-09-14T23:26:43.518506+0000","last_fullsized":"2024-09-14T23:26:43.518506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:26.921519+0000","last_clean_scrub_stamp":"2024-09-14T22:49:26.921519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:33:43.576216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044691000000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":907,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.519178+0000","last_change":"2024-09-14T22:49:22.921000+0000","last_active":"2024-09-14T23:26:43.519178+0000","last_peered":"2024-09-14T23:26:43.519178+0000","last_clean":"2024-09-14T23:26:43.519178+0000","last_became_active":"2024-09-14T22:49:22.919993+0000","last_became_peered":"2024-09-14T22:49:22.919993+0000","last_unstale":"2024-09-14T23:26:43.519178+0000","last_undegraded":"2024-09-14T23:26:43.519178+0000","last_fullsized":"2024-09-14T23:26:43.519178+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:49:21.912115+0000","last_clean_scrub_stamp":"2024-09-14T22:49:21.912115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:51.033680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706119,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1248508,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":1245374,"kb_avail":93123332,"statfs":{"total":96636764160,"available":95358291968,"internally_reserved":0,"allocated":3186688,"data_stored":1700792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1275263429},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706119,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1588088,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1584894,"kb_avail":92783752,"statfs":{"total":96636764160,"available":95010562048,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1622931910},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1312764,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1309566,"kb_avail":93059076,"statfs":{"total":96636764160,"available":95292493824,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1340996038},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:26:54.653 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-14T22:49:26.921519+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=49, tm_sec=26, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=26, tm_sec=52, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:26:54.653 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-14T22:49:26.921519+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=49, tm_sec=26, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=26, tm_sec=52, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:26:54.653 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-14T22:49:26.921519+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=49, tm_sec=26, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=26, tm_sec=52, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:26:54.653 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-14T22:49:26.921519+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=49, tm_sec=26, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=26, tm_sec=52, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:26:54.654 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-14T22:49:26.921519+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=49, tm_sec=26, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=26, tm_sec=52, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:26:54.654 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-14T22:49:26.921519+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=49, tm_sec=26, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=26, tm_sec=52, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:26:54.654 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-14T22:49:26.921519+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=49, tm_sec=26, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=26, tm_sec=52, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:26:54.654 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-14T22:49:26.921519+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=49, tm_sec=26, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=26, tm_sec=52, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:26:54.654 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-14T22:49:21.912115+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=49, tm_sec=21, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=26, tm_sec=52, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:26:54.654 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-14T23:27:14.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T23:27:14.941 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-14T23:27:14.942 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-14T23:27:14.954 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":1522,"stamp":"2024-09-14T23:27:13.457313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4149344,"kb_used_data":9456,"kb_used_omap":4,"kb_used_meta":4139835,"kb_avail":278966176,"statfs":{"total":289910292480,"available":285661364224,"internally_reserved":0,"allocated":9682944,"data_stored":6283150,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":4239191377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002661"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":787,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:56.718387+0000","last_change":"2024-09-14T23:26:56.718413+0000","last_active":"2024-09-14T23:26:56.718387+0000","last_peered":"2024-09-14T23:26:56.718387+0000","last_clean":"2024-09-14T23:26:56.718387+0000","last_became_active":"2024-09-14T22:49:27.947764+0000","last_became_peered":"2024-09-14T22:49:27.947764+0000","last_unstale":"2024-09-14T23:26:56.718387+0000","last_undegraded":"2024-09-14T23:26:56.718387+0000","last_fullsized":"2024-09-14T23:26:56.718387+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:26:56.718289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:26:56.718289+0000","last_clean_scrub_stamp":"2024-09-14T23:26:56.718289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:45:10.238493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096834600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":787,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:57.719998+0000","last_change":"2024-09-14T23:26:57.720052+0000","last_active":"2024-09-14T23:26:57.719998+0000","last_peered":"2024-09-14T23:26:57.719998+0000","last_clean":"2024-09-14T23:26:57.719998+0000","last_became_active":"2024-09-14T22:49:27.948873+0000","last_became_peered":"2024-09-14T22:49:27.948873+0000","last_unstale":"2024-09-14T23:26:57.719998+0000","last_undegraded":"2024-09-14T23:26:57.719998+0000","last_fullsized":"2024-09-14T23:26:57.719998+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:26:57.719883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:26:57.719883+0000","last_clean_scrub_stamp":"2024-09-14T23:26:57.719883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:34.632986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069214500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":788,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:58.679532+0000","last_change":"2024-09-14T23:26:58.679532+0000","last_active":"2024-09-14T23:26:58.679532+0000","last_peered":"2024-09-14T23:26:58.679532+0000","last_clean":"2024-09-14T23:26:58.679532+0000","last_became_active":"2024-09-14T22:49:27.949153+0000","last_became_peered":"2024-09-14T22:49:27.949153+0000","last_unstale":"2024-09-14T23:26:58.679532+0000","last_undegraded":"2024-09-14T23:26:58.679532+0000","last_fullsized":"2024-09-14T23:26:58.679532+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:26:58.679350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:26:58.679350+0000","last_clean_scrub_stamp":"2024-09-14T23:26:58.679350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:03:18.161795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000710178,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":788,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:59.670374+0000","last_change":"2024-09-14T23:26:59.670374+0000","last_active":"2024-09-14T23:26:59.670374+0000","last_peered":"2024-09-14T23:26:59.670374+0000","last_clean":"2024-09-14T23:26:59.670374+0000","last_became_active":"2024-09-14T22:49:27.947823+0000","last_became_peered":"2024-09-14T22:49:27.947823+0000","last_unstale":"2024-09-14T23:26:59.670374+0000","last_undegraded":"2024-09-14T23:26:59.670374+0000","last_fullsized":"2024-09-14T23:26:59.670374+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:26:59.670239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:26:59.670239+0000","last_clean_scrub_stamp":"2024-09-14T23:26:59.670239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:04:27.022578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043000900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"387'3","reported_seq":790,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:53.666600+0000","last_change":"2024-09-14T23:26:53.666600+0000","last_active":"2024-09-14T23:26:53.666600+0000","last_peered":"2024-09-14T23:26:53.666600+0000","last_clean":"2024-09-14T23:26:53.666600+0000","last_became_active":"2024-09-14T22:49:27.948120+0000","last_became_peered":"2024-09-14T22:49:27.948120+0000","last_unstale":"2024-09-14T23:26:53.666600+0000","last_undegraded":"2024-09-14T23:26:53.666600+0000","last_fullsized":"2024-09-14T23:26:53.666600+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'3","last_scrub_stamp":"2024-09-14T23:26:53.666505+0000","last_deep_scrub":"387'3","last_deep_scrub_stamp":"2024-09-14T23:26:53.666505+0000","last_clean_scrub_stamp":"2024-09-14T23:26:53.666505+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T00:24:18.374859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00095602899999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":787,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:54.657599+0000","last_change":"2024-09-14T23:26:54.657599+0000","last_active":"2024-09-14T23:26:54.657599+0000","last_peered":"2024-09-14T23:26:54.657599+0000","last_clean":"2024-09-14T23:26:54.657599+0000","last_became_active":"2024-09-14T22:49:27.948225+0000","last_became_peered":"2024-09-14T22:49:27.948225+0000","last_unstale":"2024-09-14T23:26:54.657599+0000","last_undegraded":"2024-09-14T23:26:54.657599+0000","last_fullsized":"2024-09-14T23:26:54.657599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:26:54.657497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:26:54.657497+0000","last_clean_scrub_stamp":"2024-09-14T23:26:54.657497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:06:50.339097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044858000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":787,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:55.690173+0000","last_change":"2024-09-14T23:26:55.690214+0000","last_active":"2024-09-14T23:26:55.690173+0000","last_peered":"2024-09-14T23:26:55.690173+0000","last_clean":"2024-09-14T23:26:55.690173+0000","last_became_active":"2024-09-14T22:49:27.947929+0000","last_became_peered":"2024-09-14T22:49:27.947929+0000","last_unstale":"2024-09-14T23:26:55.690173+0000","last_undegraded":"2024-09-14T23:26:55.690173+0000","last_fullsized":"2024-09-14T23:26:55.690173+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:26:55.690085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:26:55.690085+0000","last_clean_scrub_stamp":"2024-09-14T23:26:55.690085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:34:04.695404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"387'2","reported_seq":790,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:55.688919+0000","last_change":"2024-09-14T23:26:55.688919+0000","last_active":"2024-09-14T23:26:55.688919+0000","last_peered":"2024-09-14T23:26:55.688919+0000","last_clean":"2024-09-14T23:26:55.688919+0000","last_became_active":"2024-09-14T22:49:27.947559+0000","last_became_peered":"2024-09-14T22:49:27.947559+0000","last_unstale":"2024-09-14T23:26:55.688919+0000","last_undegraded":"2024-09-14T23:26:55.688919+0000","last_fullsized":"2024-09-14T23:26:55.688919+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'2","last_scrub_stamp":"2024-09-14T23:26:55.688801+0000","last_deep_scrub":"387'2","last_deep_scrub_stamp":"2024-09-14T23:26:55.688801+0000","last_clean_scrub_stamp":"2024-09-14T23:26:55.688801+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:24:56.163293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044691000000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"387'99","reported_seq":918,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-14T23:26:54.672326+0000","last_change":"2024-09-14T23:26:54.672355+0000","last_active":"2024-09-14T23:26:54.672326+0000","last_peered":"2024-09-14T23:26:54.672326+0000","last_clean":"2024-09-14T23:26:54.672326+0000","last_became_active":"2024-09-14T22:49:22.919993+0000","last_became_peered":"2024-09-14T22:49:22.919993+0000","last_unstale":"2024-09-14T23:26:54.672326+0000","last_undegraded":"2024-09-14T23:26:54.672326+0000","last_fullsized":"2024-09-14T23:26:54.672326+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'99","last_scrub_stamp":"2024-09-14T23:26:54.672245+0000","last_deep_scrub":"387'99","last_deep_scrub_stamp":"2024-09-14T23:26:54.672245+0000","last_clean_scrub_stamp":"2024-09-14T23:26:54.672245+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:37:00.278642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5798,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706123,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1248504,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":1245374,"kb_avail":93123336,"statfs":{"total":96636764160,"available":95358296064,"internally_reserved":0,"allocated":3186688,"data_stored":1700792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1275263429},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706123,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1588084,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1584894,"kb_avail":92783756,"statfs":{"total":96636764160,"available":95010566144,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1622931910},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1312756,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1309566,"kb_avail":93059084,"statfs":{"total":96636764160,"available":95292502016,"internally_reserved":0,"allocated":3248128,"data_stored":2291179,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1340996038},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2899,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:27:14.955 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-14T23:27:15.243 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-14T23:27:15.243 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-14T23:27:15.243 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-14T23:27:15.243 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:15.313 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-14T23:27:15.313 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-14T23:27:15.313 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:15.398 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-14T23:27:15.399 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-14T23:27:15.399 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:15.474 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-14T23:27:15.474 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-14T23:27:15.475 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-14T23:27:15.475 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:15.505 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-14T23:27:15.505 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-14T23:27:15.505 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-14T23:27:15.505 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:15.518 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-14T23:27:15.518 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-14T23:27:15.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T23:27:15.648 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T23:27:15.648 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-14T23:27:15.998 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T23:27:15.998 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-14T23:27:16.130 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T23:27:16.130 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-14T23:27:16.247 INFO:tasks.ceph:Archiving mon data... 2024-09-14T23:27:16.248 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733/data/cluster2.mon.a.tgz 2024-09-14T23:27:16.248 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-09-14T23:27:16.266 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.q5c1kJoujL 2024-09-14T23:27:16.266 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.q5c1kJoujL 2024-09-14T23:27:16.436 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.q5c1kJoujL 2024-09-14T23:27:16.556 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.q5c1kJoujL is 421KB 2024-09-14T23:27:16.654 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.q5c1kJoujL 2024-09-14T23:27:16.670 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-14T23:27:16.671 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T23:27:16.674 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T23:27:16.738 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-14T23:27:16.753 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-14T23:27:16.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T23:27:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T23:27:17.054 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-14T23:27:17.066 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":1618,"stamp":"2024-09-14T23:27:16.238739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":297704,"kb_used_data":11304,"kb_used_omap":4,"kb_used_meta":286331,"kb_avail":282817816,"statfs":{"total":289910292480,"available":289605443584,"internally_reserved":0,"allocated":11575296,"data_stored":7308358,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":293203279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":932,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518222+0000","last_change":"2024-09-14T22:48:51.757209+0000","last_active":"2024-09-14T23:26:44.518222+0000","last_peered":"2024-09-14T23:26:44.518222+0000","last_clean":"2024-09-14T23:26:44.518222+0000","last_became_active":"2024-09-14T22:48:49.753482+0000","last_became_peered":"2024-09-14T22:48:49.753482+0000","last_unstale":"2024-09-14T23:26:44.518222+0000","last_undegraded":"2024-09-14T23:26:44.518222+0000","last_fullsized":"2024-09-14T23:26:44.518222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:13:32.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011426839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":932,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518223+0000","last_change":"2024-09-14T22:48:51.756962+0000","last_active":"2024-09-14T23:26:44.518223+0000","last_peered":"2024-09-14T23:26:44.518223+0000","last_clean":"2024-09-14T23:26:44.518223+0000","last_became_active":"2024-09-14T22:48:49.753595+0000","last_became_peered":"2024-09-14T22:48:49.753595+0000","last_unstale":"2024-09-14T23:26:44.518223+0000","last_undegraded":"2024-09-14T23:26:44.518223+0000","last_fullsized":"2024-09-14T23:26:44.518223+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.951246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091631000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.520210+0000","last_change":"2024-09-14T22:48:51.756913+0000","last_active":"2024-09-14T23:26:44.520210+0000","last_peered":"2024-09-14T23:26:44.520210+0000","last_clean":"2024-09-14T23:26:44.520210+0000","last_became_active":"2024-09-14T22:48:49.753420+0000","last_became_peered":"2024-09-14T22:48:49.753420+0000","last_unstale":"2024-09-14T23:26:44.520210+0000","last_undegraded":"2024-09-14T23:26:44.520210+0000","last_fullsized":"2024-09-14T23:26:44.520210+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:38.693996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086764700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518815+0000","last_change":"2024-09-14T22:48:51.756569+0000","last_active":"2024-09-14T23:26:44.518815+0000","last_peered":"2024-09-14T23:26:44.518815+0000","last_clean":"2024-09-14T23:26:44.518815+0000","last_became_active":"2024-09-14T22:48:49.752095+0000","last_became_peered":"2024-09-14T22:48:49.752095+0000","last_unstale":"2024-09-14T23:26:44.518815+0000","last_undegraded":"2024-09-14T23:26:44.518815+0000","last_fullsized":"2024-09-14T23:26:44.518815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:37:07.776688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054311300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:42.536759+0000","last_change":"2024-09-14T22:48:51.753263+0000","last_active":"2024-09-14T23:26:42.536759+0000","last_peered":"2024-09-14T23:26:42.536759+0000","last_clean":"2024-09-14T23:26:42.536759+0000","last_became_active":"2024-09-14T22:48:49.752329+0000","last_became_peered":"2024-09-14T22:48:49.752329+0000","last_unstale":"2024-09-14T23:26:42.536759+0000","last_undegraded":"2024-09-14T23:26:42.536759+0000","last_fullsized":"2024-09-14T23:26:42.536759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:34:30.769014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039893099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.882022+0000","last_change":"2024-09-14T22:48:51.803183+0000","last_active":"2024-09-14T23:26:43.882022+0000","last_peered":"2024-09-14T23:26:43.882022+0000","last_clean":"2024-09-14T23:26:43.882022+0000","last_became_active":"2024-09-14T22:48:49.752614+0000","last_became_peered":"2024-09-14T22:48:49.752614+0000","last_unstale":"2024-09-14T23:26:43.882022+0000","last_undegraded":"2024-09-14T23:26:43.882022+0000","last_fullsized":"2024-09-14T23:26:43.882022+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:04:25.567425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045173899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.883793+0000","last_change":"2024-09-14T22:48:51.803192+0000","last_active":"2024-09-14T23:26:43.883793+0000","last_peered":"2024-09-14T23:26:43.883793+0000","last_clean":"2024-09-14T23:26:43.883793+0000","last_became_active":"2024-09-14T22:48:49.753296+0000","last_became_peered":"2024-09-14T22:48:49.753296+0000","last_unstale":"2024-09-14T23:26:43.883793+0000","last_undegraded":"2024-09-14T23:26:43.883793+0000","last_fullsized":"2024-09-14T23:26:43.883793+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T22:50:00.939574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045554200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.521411+0000","last_change":"2024-09-14T22:48:51.756500+0000","last_active":"2024-09-14T23:26:44.521411+0000","last_peered":"2024-09-14T23:26:44.521411+0000","last_clean":"2024-09-14T23:26:44.521411+0000","last_became_active":"2024-09-14T22:48:49.752332+0000","last_became_peered":"2024-09-14T22:48:49.752332+0000","last_unstale":"2024-09-14T23:26:44.521411+0000","last_undegraded":"2024-09-14T23:26:44.521411+0000","last_fullsized":"2024-09-14T23:26:44.521411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:11:52.254597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061883999999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1062,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.521119+0000","last_change":"2024-09-14T22:48:46.321941+0000","last_active":"2024-09-14T23:26:44.521119+0000","last_peered":"2024-09-14T23:26:44.521119+0000","last_clean":"2024-09-14T23:26:44.521119+0000","last_became_active":"2024-09-14T22:48:46.320719+0000","last_became_peered":"2024-09-14T22:48:46.320719+0000","last_unstale":"2024-09-14T23:26:44.521119+0000","last_undegraded":"2024-09-14T23:26:44.521119+0000","last_fullsized":"2024-09-14T23:26:44.521119+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_clean_scrub_stamp":"2024-09-14T22:48:44.727263+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:47.941296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97316,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274524,"statfs":{"total":96636764160,"available":96537112576,"internally_reserved":0,"allocated":3817472,"data_stored":2042528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75232,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296608,"statfs":{"total":96636764160,"available":96559726592,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125156,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246684,"statfs":{"total":96636764160,"available":96508604416,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:27:17.067 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T23:27:17.353 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T23:27:17.353 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-14T23:27:17.365 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":1618,"stamp":"2024-09-14T23:27:16.238739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":297704,"kb_used_data":11304,"kb_used_omap":4,"kb_used_meta":286331,"kb_avail":282817816,"statfs":{"total":289910292480,"available":289605443584,"internally_reserved":0,"allocated":11575296,"data_stored":7308358,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":293203279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":932,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518222+0000","last_change":"2024-09-14T22:48:51.757209+0000","last_active":"2024-09-14T23:26:44.518222+0000","last_peered":"2024-09-14T23:26:44.518222+0000","last_clean":"2024-09-14T23:26:44.518222+0000","last_became_active":"2024-09-14T22:48:49.753482+0000","last_became_peered":"2024-09-14T22:48:49.753482+0000","last_unstale":"2024-09-14T23:26:44.518222+0000","last_undegraded":"2024-09-14T23:26:44.518222+0000","last_fullsized":"2024-09-14T23:26:44.518222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:13:32.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011426839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":932,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518223+0000","last_change":"2024-09-14T22:48:51.756962+0000","last_active":"2024-09-14T23:26:44.518223+0000","last_peered":"2024-09-14T23:26:44.518223+0000","last_clean":"2024-09-14T23:26:44.518223+0000","last_became_active":"2024-09-14T22:48:49.753595+0000","last_became_peered":"2024-09-14T22:48:49.753595+0000","last_unstale":"2024-09-14T23:26:44.518223+0000","last_undegraded":"2024-09-14T23:26:44.518223+0000","last_fullsized":"2024-09-14T23:26:44.518223+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.951246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091631000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.520210+0000","last_change":"2024-09-14T22:48:51.756913+0000","last_active":"2024-09-14T23:26:44.520210+0000","last_peered":"2024-09-14T23:26:44.520210+0000","last_clean":"2024-09-14T23:26:44.520210+0000","last_became_active":"2024-09-14T22:48:49.753420+0000","last_became_peered":"2024-09-14T22:48:49.753420+0000","last_unstale":"2024-09-14T23:26:44.520210+0000","last_undegraded":"2024-09-14T23:26:44.520210+0000","last_fullsized":"2024-09-14T23:26:44.520210+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:38.693996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086764700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518815+0000","last_change":"2024-09-14T22:48:51.756569+0000","last_active":"2024-09-14T23:26:44.518815+0000","last_peered":"2024-09-14T23:26:44.518815+0000","last_clean":"2024-09-14T23:26:44.518815+0000","last_became_active":"2024-09-14T22:48:49.752095+0000","last_became_peered":"2024-09-14T22:48:49.752095+0000","last_unstale":"2024-09-14T23:26:44.518815+0000","last_undegraded":"2024-09-14T23:26:44.518815+0000","last_fullsized":"2024-09-14T23:26:44.518815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:37:07.776688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054311300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:42.536759+0000","last_change":"2024-09-14T22:48:51.753263+0000","last_active":"2024-09-14T23:26:42.536759+0000","last_peered":"2024-09-14T23:26:42.536759+0000","last_clean":"2024-09-14T23:26:42.536759+0000","last_became_active":"2024-09-14T22:48:49.752329+0000","last_became_peered":"2024-09-14T22:48:49.752329+0000","last_unstale":"2024-09-14T23:26:42.536759+0000","last_undegraded":"2024-09-14T23:26:42.536759+0000","last_fullsized":"2024-09-14T23:26:42.536759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:34:30.769014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039893099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.882022+0000","last_change":"2024-09-14T22:48:51.803183+0000","last_active":"2024-09-14T23:26:43.882022+0000","last_peered":"2024-09-14T23:26:43.882022+0000","last_clean":"2024-09-14T23:26:43.882022+0000","last_became_active":"2024-09-14T22:48:49.752614+0000","last_became_peered":"2024-09-14T22:48:49.752614+0000","last_unstale":"2024-09-14T23:26:43.882022+0000","last_undegraded":"2024-09-14T23:26:43.882022+0000","last_fullsized":"2024-09-14T23:26:43.882022+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:04:25.567425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045173899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.883793+0000","last_change":"2024-09-14T22:48:51.803192+0000","last_active":"2024-09-14T23:26:43.883793+0000","last_peered":"2024-09-14T23:26:43.883793+0000","last_clean":"2024-09-14T23:26:43.883793+0000","last_became_active":"2024-09-14T22:48:49.753296+0000","last_became_peered":"2024-09-14T22:48:49.753296+0000","last_unstale":"2024-09-14T23:26:43.883793+0000","last_undegraded":"2024-09-14T23:26:43.883793+0000","last_fullsized":"2024-09-14T23:26:43.883793+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T22:50:00.939574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045554200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.521411+0000","last_change":"2024-09-14T22:48:51.756500+0000","last_active":"2024-09-14T23:26:44.521411+0000","last_peered":"2024-09-14T23:26:44.521411+0000","last_clean":"2024-09-14T23:26:44.521411+0000","last_became_active":"2024-09-14T22:48:49.752332+0000","last_became_peered":"2024-09-14T22:48:49.752332+0000","last_unstale":"2024-09-14T23:26:44.521411+0000","last_undegraded":"2024-09-14T23:26:44.521411+0000","last_fullsized":"2024-09-14T23:26:44.521411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:11:52.254597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061883999999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1062,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.521119+0000","last_change":"2024-09-14T22:48:46.321941+0000","last_active":"2024-09-14T23:26:44.521119+0000","last_peered":"2024-09-14T23:26:44.521119+0000","last_clean":"2024-09-14T23:26:44.521119+0000","last_became_active":"2024-09-14T22:48:46.320719+0000","last_became_peered":"2024-09-14T22:48:46.320719+0000","last_unstale":"2024-09-14T23:26:44.521119+0000","last_undegraded":"2024-09-14T23:26:44.521119+0000","last_fullsized":"2024-09-14T23:26:44.521119+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_clean_scrub_stamp":"2024-09-14T22:48:44.727263+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:47.941296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97316,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274524,"statfs":{"total":96636764160,"available":96537112576,"internally_reserved":0,"allocated":3817472,"data_stored":2042528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75232,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296608,"statfs":{"total":96636764160,"available":96559726592,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125156,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246684,"statfs":{"total":96636764160,"available":96508604416,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:27:17.366 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-14T23:27:17.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T23:27:17.651 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T23:27:17.652 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-14T23:27:17.664 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":1618,"stamp":"2024-09-14T23:27:16.238739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":297704,"kb_used_data":11304,"kb_used_omap":4,"kb_used_meta":286331,"kb_avail":282817816,"statfs":{"total":289910292480,"available":289605443584,"internally_reserved":0,"allocated":11575296,"data_stored":7308358,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":293203279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":932,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518222+0000","last_change":"2024-09-14T22:48:51.757209+0000","last_active":"2024-09-14T23:26:44.518222+0000","last_peered":"2024-09-14T23:26:44.518222+0000","last_clean":"2024-09-14T23:26:44.518222+0000","last_became_active":"2024-09-14T22:48:49.753482+0000","last_became_peered":"2024-09-14T22:48:49.753482+0000","last_unstale":"2024-09-14T23:26:44.518222+0000","last_undegraded":"2024-09-14T23:26:44.518222+0000","last_fullsized":"2024-09-14T23:26:44.518222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:13:32.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011426839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":932,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518223+0000","last_change":"2024-09-14T22:48:51.756962+0000","last_active":"2024-09-14T23:26:44.518223+0000","last_peered":"2024-09-14T23:26:44.518223+0000","last_clean":"2024-09-14T23:26:44.518223+0000","last_became_active":"2024-09-14T22:48:49.753595+0000","last_became_peered":"2024-09-14T22:48:49.753595+0000","last_unstale":"2024-09-14T23:26:44.518223+0000","last_undegraded":"2024-09-14T23:26:44.518223+0000","last_fullsized":"2024-09-14T23:26:44.518223+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.951246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091631000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.520210+0000","last_change":"2024-09-14T22:48:51.756913+0000","last_active":"2024-09-14T23:26:44.520210+0000","last_peered":"2024-09-14T23:26:44.520210+0000","last_clean":"2024-09-14T23:26:44.520210+0000","last_became_active":"2024-09-14T22:48:49.753420+0000","last_became_peered":"2024-09-14T22:48:49.753420+0000","last_unstale":"2024-09-14T23:26:44.520210+0000","last_undegraded":"2024-09-14T23:26:44.520210+0000","last_fullsized":"2024-09-14T23:26:44.520210+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:38.693996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086764700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518815+0000","last_change":"2024-09-14T22:48:51.756569+0000","last_active":"2024-09-14T23:26:44.518815+0000","last_peered":"2024-09-14T23:26:44.518815+0000","last_clean":"2024-09-14T23:26:44.518815+0000","last_became_active":"2024-09-14T22:48:49.752095+0000","last_became_peered":"2024-09-14T22:48:49.752095+0000","last_unstale":"2024-09-14T23:26:44.518815+0000","last_undegraded":"2024-09-14T23:26:44.518815+0000","last_fullsized":"2024-09-14T23:26:44.518815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:37:07.776688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054311300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:42.536759+0000","last_change":"2024-09-14T22:48:51.753263+0000","last_active":"2024-09-14T23:26:42.536759+0000","last_peered":"2024-09-14T23:26:42.536759+0000","last_clean":"2024-09-14T23:26:42.536759+0000","last_became_active":"2024-09-14T22:48:49.752329+0000","last_became_peered":"2024-09-14T22:48:49.752329+0000","last_unstale":"2024-09-14T23:26:42.536759+0000","last_undegraded":"2024-09-14T23:26:42.536759+0000","last_fullsized":"2024-09-14T23:26:42.536759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:34:30.769014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039893099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.882022+0000","last_change":"2024-09-14T22:48:51.803183+0000","last_active":"2024-09-14T23:26:43.882022+0000","last_peered":"2024-09-14T23:26:43.882022+0000","last_clean":"2024-09-14T23:26:43.882022+0000","last_became_active":"2024-09-14T22:48:49.752614+0000","last_became_peered":"2024-09-14T22:48:49.752614+0000","last_unstale":"2024-09-14T23:26:43.882022+0000","last_undegraded":"2024-09-14T23:26:43.882022+0000","last_fullsized":"2024-09-14T23:26:43.882022+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:04:25.567425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045173899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.883793+0000","last_change":"2024-09-14T22:48:51.803192+0000","last_active":"2024-09-14T23:26:43.883793+0000","last_peered":"2024-09-14T23:26:43.883793+0000","last_clean":"2024-09-14T23:26:43.883793+0000","last_became_active":"2024-09-14T22:48:49.753296+0000","last_became_peered":"2024-09-14T22:48:49.753296+0000","last_unstale":"2024-09-14T23:26:43.883793+0000","last_undegraded":"2024-09-14T23:26:43.883793+0000","last_fullsized":"2024-09-14T23:26:43.883793+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T22:50:00.939574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045554200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.521411+0000","last_change":"2024-09-14T22:48:51.756500+0000","last_active":"2024-09-14T23:26:44.521411+0000","last_peered":"2024-09-14T23:26:44.521411+0000","last_clean":"2024-09-14T23:26:44.521411+0000","last_became_active":"2024-09-14T22:48:49.752332+0000","last_became_peered":"2024-09-14T22:48:49.752332+0000","last_unstale":"2024-09-14T23:26:44.521411+0000","last_undegraded":"2024-09-14T23:26:44.521411+0000","last_fullsized":"2024-09-14T23:26:44.521411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:11:52.254597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061883999999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1062,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.521119+0000","last_change":"2024-09-14T22:48:46.321941+0000","last_active":"2024-09-14T23:26:44.521119+0000","last_peered":"2024-09-14T23:26:44.521119+0000","last_clean":"2024-09-14T23:26:44.521119+0000","last_became_active":"2024-09-14T22:48:46.320719+0000","last_became_peered":"2024-09-14T22:48:46.320719+0000","last_unstale":"2024-09-14T23:26:44.521119+0000","last_undegraded":"2024-09-14T23:26:44.521119+0000","last_fullsized":"2024-09-14T23:26:44.521119+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_clean_scrub_stamp":"2024-09-14T22:48:44.727263+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:47.941296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97316,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274524,"statfs":{"total":96636764160,"available":96537112576,"internally_reserved":0,"allocated":3817472,"data_stored":2042528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75232,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296608,"statfs":{"total":96636764160,"available":96559726592,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125156,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246684,"statfs":{"total":96636764160,"available":96508604416,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:27:17.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T23:27:17.955 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T23:27:17.955 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":464,"fsid":"51bdee54-6793-4abd-9100-af548f45d456","created":"2024-09-14T22:48:33.177127+0000","modified":"2024-09-14T23:26:42.527416+0000","last_up_change":"2024-09-14T22:48:42.725961+0000","last_in_change":"2024-09-14T22:48:34.071469+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T22:48:43.862770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T22:48:48.717572+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d0cb2c92-7fae-48c0-9a16-dd219ce47a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6809","nonce":2942497514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6811","nonce":2942497514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6815","nonce":2942497514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2942497514},{"type":"v1","addr":"172.21.15.52:6813","nonce":2942497514}]},"public_addr":"172.21.15.52:6809/2942497514","cluster_addr":"172.21.15.52:6811/2942497514","heartbeat_back_addr":"172.21.15.52:6815/2942497514","heartbeat_front_addr":"172.21.15.52:6813/2942497514","state":["exists","up"]},{"osd":1,"uuid":"69ac61b7-3004-4643-b6f8-280f99d135ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6817","nonce":881972827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6819","nonce":881972827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6823","nonce":881972827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":881972827},{"type":"v1","addr":"172.21.15.52:6821","nonce":881972827}]},"public_addr":"172.21.15.52:6817/881972827","cluster_addr":"172.21.15.52:6819/881972827","heartbeat_back_addr":"172.21.15.52:6823/881972827","heartbeat_front_addr":"172.21.15.52:6821/881972827","state":["exists","up"]},{"osd":2,"uuid":"eb017cba-c46f-4e11-8c91-f395956b7643","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6801","nonce":2826339032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6803","nonce":2826339032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6807","nonce":2826339032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":2826339032},{"type":"v1","addr":"172.21.15.52:6805","nonce":2826339032}]},"public_addr":"172.21.15.52:6801/2826339032","cluster_addr":"172.21.15.52:6803/2826339032","heartbeat_back_addr":"172.21.15.52:6807/2826339032","heartbeat_front_addr":"172.21.15.52:6805/2826339032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:40.378456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:40.126245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T22:48:39.874970+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T23:27:18.969 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-14T23:27:18.969 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-14T23:27:19.109 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-14T23:27:19.109 INFO:teuthology.orchestra.run.smithi052.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T23:27:19.109 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-14T23:27:19.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-14T23:27:19.414 INFO:teuthology.orchestra.run.smithi052.stderr:instructed osd(s) 0 to deep-scrub 2024-09-14T23:27:19.426 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-14T23:27:19.427 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-14T23:27:19.552 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-14T23:27:19.552 INFO:teuthology.orchestra.run.smithi052.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T23:27:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-14T23:27:19.563 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-14T23:27:19.852 INFO:teuthology.orchestra.run.smithi052.stderr:instructed osd(s) 1 to deep-scrub 2024-09-14T23:27:19.865 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-14T23:27:19.865 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-14T23:27:19.991 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-14T23:27:19.991 INFO:teuthology.orchestra.run.smithi052.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T23:27:19.991 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-14T23:27:20.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-14T23:27:20.288 INFO:teuthology.orchestra.run.smithi052.stderr:instructed osd(s) 2 to deep-scrub 2024-09-14T23:27:20.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T23:27:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T23:27:20.591 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-14T23:27:20.603 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2024-09-14T23:27:20.239460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":297704,"kb_used_data":11304,"kb_used_omap":4,"kb_used_meta":286331,"kb_avail":282817816,"statfs":{"total":289910292480,"available":289605443584,"internally_reserved":0,"allocated":11575296,"data_stored":7308358,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":293203279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002438"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":932,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518222+0000","last_change":"2024-09-14T22:48:51.757209+0000","last_active":"2024-09-14T23:26:44.518222+0000","last_peered":"2024-09-14T23:26:44.518222+0000","last_clean":"2024-09-14T23:26:44.518222+0000","last_became_active":"2024-09-14T22:48:49.753482+0000","last_became_peered":"2024-09-14T22:48:49.753482+0000","last_unstale":"2024-09-14T23:26:44.518222+0000","last_undegraded":"2024-09-14T23:26:44.518222+0000","last_fullsized":"2024-09-14T23:26:44.518222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T01:13:32.670337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011426839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":932,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518223+0000","last_change":"2024-09-14T22:48:51.756962+0000","last_active":"2024-09-14T23:26:44.518223+0000","last_peered":"2024-09-14T23:26:44.518223+0000","last_clean":"2024-09-14T23:26:44.518223+0000","last_became_active":"2024-09-14T22:48:49.753595+0000","last_became_peered":"2024-09-14T22:48:49.753595+0000","last_unstale":"2024-09-14T23:26:44.518223+0000","last_undegraded":"2024-09-14T23:26:44.518223+0000","last_fullsized":"2024-09-14T23:26:44.518223+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.951246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091631000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.520210+0000","last_change":"2024-09-14T22:48:51.756913+0000","last_active":"2024-09-14T23:26:44.520210+0000","last_peered":"2024-09-14T23:26:44.520210+0000","last_clean":"2024-09-14T23:26:44.520210+0000","last_became_active":"2024-09-14T22:48:49.753420+0000","last_became_peered":"2024-09-14T22:48:49.753420+0000","last_unstale":"2024-09-14T23:26:44.520210+0000","last_undegraded":"2024-09-14T23:26:44.520210+0000","last_fullsized":"2024-09-14T23:26:44.520210+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:38.693996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086764700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.518815+0000","last_change":"2024-09-14T22:48:51.756569+0000","last_active":"2024-09-14T23:26:44.518815+0000","last_peered":"2024-09-14T23:26:44.518815+0000","last_clean":"2024-09-14T23:26:44.518815+0000","last_became_active":"2024-09-14T22:48:49.752095+0000","last_became_peered":"2024-09-14T22:48:49.752095+0000","last_unstale":"2024-09-14T23:26:44.518815+0000","last_undegraded":"2024-09-14T23:26:44.518815+0000","last_fullsized":"2024-09-14T23:26:44.518815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:37:07.776688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054311300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:42.536759+0000","last_change":"2024-09-14T22:48:51.753263+0000","last_active":"2024-09-14T23:26:42.536759+0000","last_peered":"2024-09-14T23:26:42.536759+0000","last_clean":"2024-09-14T23:26:42.536759+0000","last_became_active":"2024-09-14T22:48:49.752329+0000","last_became_peered":"2024-09-14T22:48:49.752329+0000","last_unstale":"2024-09-14T23:26:42.536759+0000","last_undegraded":"2024-09-14T23:26:42.536759+0000","last_fullsized":"2024-09-14T23:26:42.536759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:34:30.769014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039893099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":933,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.882022+0000","last_change":"2024-09-14T22:48:51.803183+0000","last_active":"2024-09-14T23:26:43.882022+0000","last_peered":"2024-09-14T23:26:43.882022+0000","last_clean":"2024-09-14T23:26:43.882022+0000","last_became_active":"2024-09-14T22:48:49.752614+0000","last_became_peered":"2024-09-14T22:48:49.752614+0000","last_unstale":"2024-09-14T23:26:43.882022+0000","last_undegraded":"2024-09-14T23:26:43.882022+0000","last_fullsized":"2024-09-14T23:26:43.882022+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:04:25.567425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045173899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:43.883793+0000","last_change":"2024-09-14T22:48:51.803192+0000","last_active":"2024-09-14T23:26:43.883793+0000","last_peered":"2024-09-14T23:26:43.883793+0000","last_clean":"2024-09-14T23:26:43.883793+0000","last_became_active":"2024-09-14T22:48:49.753296+0000","last_became_peered":"2024-09-14T22:48:49.753296+0000","last_unstale":"2024-09-14T23:26:43.883793+0000","last_undegraded":"2024-09-14T23:26:43.883793+0000","last_fullsized":"2024-09-14T23:26:43.883793+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T22:50:00.939574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045554200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":934,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.521411+0000","last_change":"2024-09-14T22:48:51.756500+0000","last_active":"2024-09-14T23:26:44.521411+0000","last_peered":"2024-09-14T23:26:44.521411+0000","last_clean":"2024-09-14T23:26:44.521411+0000","last_became_active":"2024-09-14T22:48:49.752332+0000","last_became_peered":"2024-09-14T22:48:49.752332+0000","last_unstale":"2024-09-14T23:26:44.521411+0000","last_undegraded":"2024-09-14T23:26:44.521411+0000","last_fullsized":"2024-09-14T23:26:44.521411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:48.739523+0000","last_clean_scrub_stamp":"2024-09-14T22:48:48.739523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:11:52.254597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061883999999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1062,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:26:44.521119+0000","last_change":"2024-09-14T22:48:46.321941+0000","last_active":"2024-09-14T23:26:44.521119+0000","last_peered":"2024-09-14T23:26:44.521119+0000","last_clean":"2024-09-14T23:26:44.521119+0000","last_became_active":"2024-09-14T22:48:46.320719+0000","last_became_peered":"2024-09-14T22:48:46.320719+0000","last_unstale":"2024-09-14T23:26:44.521119+0000","last_undegraded":"2024-09-14T23:26:44.521119+0000","last_fullsized":"2024-09-14T23:26:44.521119+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T22:48:44.727263+0000","last_clean_scrub_stamp":"2024-09-14T22:48:44.727263+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:19:47.941296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97316,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274524,"statfs":{"total":96636764160,"available":96537112576,"internally_reserved":0,"allocated":3817472,"data_stored":2042528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75232,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296608,"statfs":{"total":96636764160,"available":96559726592,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738836,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125156,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246684,"statfs":{"total":96636764160,"available":96508604416,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:27:20.604 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-14T22:48:48.739523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=48, tm_sec=48, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:27:20.604 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-14T22:48:48.739523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=48, tm_sec=48, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:27:20.605 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-14T22:48:48.739523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=48, tm_sec=48, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:27:20.605 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-14T22:48:48.739523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=48, tm_sec=48, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:27:20.605 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-14T22:48:48.739523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=48, tm_sec=48, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:27:20.605 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-14T22:48:48.739523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=48, tm_sec=48, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:27:20.605 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-14T22:48:48.739523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=48, tm_sec=48, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:27:20.605 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-14T22:48:48.739523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=48, tm_sec=48, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:27:20.605 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-14T22:48:44.727263+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=22, tm_min=48, tm_sec=44, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=23, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T23:27:20.605 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-14T23:27:40.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T23:27:40.907 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-14T23:27:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-14T23:27:40.920 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":1630,"stamp":"2024-09-14T23:27:40.243841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":297688,"kb_used_data":11304,"kb_used_omap":4,"kb_used_meta":286331,"kb_avail":282817832,"statfs":{"total":289910292480,"available":289605459968,"internally_reserved":0,"allocated":11575296,"data_stored":7308358,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":293203279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":941,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:27:23.830931+0000","last_change":"2024-09-14T23:27:23.830958+0000","last_active":"2024-09-14T23:27:23.830931+0000","last_peered":"2024-09-14T23:27:23.830931+0000","last_clean":"2024-09-14T23:27:23.830931+0000","last_became_active":"2024-09-14T22:48:49.753482+0000","last_became_peered":"2024-09-14T22:48:49.753482+0000","last_unstale":"2024-09-14T23:27:23.830931+0000","last_undegraded":"2024-09-14T23:27:23.830931+0000","last_fullsized":"2024-09-14T23:27:23.830931+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:27:23.830807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:27:23.830807+0000","last_clean_scrub_stamp":"2024-09-14T23:27:23.830807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T23:33:07.011686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011426839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":941,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:27:24.816141+0000","last_change":"2024-09-14T23:27:24.816184+0000","last_active":"2024-09-14T23:27:24.816141+0000","last_peered":"2024-09-14T23:27:24.816141+0000","last_clean":"2024-09-14T23:27:24.816141+0000","last_became_active":"2024-09-14T22:48:49.753595+0000","last_became_peered":"2024-09-14T22:48:49.753595+0000","last_unstale":"2024-09-14T23:27:24.816141+0000","last_undegraded":"2024-09-14T23:27:24.816141+0000","last_fullsized":"2024-09-14T23:27:24.816141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:27:24.816007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:27:24.816007+0000","last_clean_scrub_stamp":"2024-09-14T23:27:24.816007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:46:14.714171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00091631000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:27:22.871953+0000","last_change":"2024-09-14T23:27:22.871981+0000","last_active":"2024-09-14T23:27:22.871953+0000","last_peered":"2024-09-14T23:27:22.871953+0000","last_clean":"2024-09-14T23:27:22.871953+0000","last_became_active":"2024-09-14T22:48:49.753420+0000","last_became_peered":"2024-09-14T22:48:49.753420+0000","last_unstale":"2024-09-14T23:27:22.871953+0000","last_undegraded":"2024-09-14T23:27:22.871953+0000","last_fullsized":"2024-09-14T23:27:22.871953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:27:22.871834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:27:22.871834+0000","last_clean_scrub_stamp":"2024-09-14T23:27:22.871834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:59:23.780010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086764700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":942,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:27:21.828525+0000","last_change":"2024-09-14T23:27:21.828552+0000","last_active":"2024-09-14T23:27:21.828525+0000","last_peered":"2024-09-14T23:27:21.828525+0000","last_clean":"2024-09-14T23:27:21.828525+0000","last_became_active":"2024-09-14T22:48:49.752095+0000","last_became_peered":"2024-09-14T22:48:49.752095+0000","last_unstale":"2024-09-14T23:27:21.828525+0000","last_undegraded":"2024-09-14T23:27:21.828525+0000","last_fullsized":"2024-09-14T23:27:21.828525+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:27:21.828414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:27:21.828414+0000","last_clean_scrub_stamp":"2024-09-14T23:27:21.828414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:45.048721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054311300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"464'3","reported_seq":944,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:27:19.936289+0000","last_change":"2024-09-14T23:27:19.936336+0000","last_active":"2024-09-14T23:27:19.936289+0000","last_peered":"2024-09-14T23:27:19.936289+0000","last_clean":"2024-09-14T23:27:19.936289+0000","last_became_active":"2024-09-14T22:48:49.752329+0000","last_became_peered":"2024-09-14T22:48:49.752329+0000","last_unstale":"2024-09-14T23:27:19.936289+0000","last_undegraded":"2024-09-14T23:27:19.936289+0000","last_fullsized":"2024-09-14T23:27:19.936289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'3","last_scrub_stamp":"2024-09-14T23:27:19.936195+0000","last_deep_scrub":"464'3","last_deep_scrub_stamp":"2024-09-14T23:27:19.936195+0000","last_clean_scrub_stamp":"2024-09-14T23:27:19.936195+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:04:24.881920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00039893099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":942,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:27:20.995286+0000","last_change":"2024-09-14T23:27:20.995327+0000","last_active":"2024-09-14T23:27:20.995286+0000","last_peered":"2024-09-14T23:27:20.995286+0000","last_clean":"2024-09-14T23:27:20.995286+0000","last_became_active":"2024-09-14T22:48:49.752614+0000","last_became_peered":"2024-09-14T22:48:49.752614+0000","last_unstale":"2024-09-14T23:27:20.995286+0000","last_undegraded":"2024-09-14T23:27:20.995286+0000","last_fullsized":"2024-09-14T23:27:20.995286+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:27:20.995187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:27:20.995187+0000","last_clean_scrub_stamp":"2024-09-14T23:27:20.995187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:58:54.861484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045173899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:27:21.955307+0000","last_change":"2024-09-14T23:27:21.955336+0000","last_active":"2024-09-14T23:27:21.955307+0000","last_peered":"2024-09-14T23:27:21.955307+0000","last_clean":"2024-09-14T23:27:21.955307+0000","last_became_active":"2024-09-14T22:48:49.753296+0000","last_became_peered":"2024-09-14T22:48:49.753296+0000","last_unstale":"2024-09-14T23:27:21.955307+0000","last_undegraded":"2024-09-14T23:27:21.955307+0000","last_fullsized":"2024-09-14T23:27:21.955307+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T23:27:21.955211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T23:27:21.955211+0000","last_clean_scrub_stamp":"2024-09-14T23:27:21.955211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T00:31:47.283030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045554200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"464'2","reported_seq":944,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:27:25.804380+0000","last_change":"2024-09-14T23:27:25.804407+0000","last_active":"2024-09-14T23:27:25.804380+0000","last_peered":"2024-09-14T23:27:25.804380+0000","last_clean":"2024-09-14T23:27:25.804380+0000","last_became_active":"2024-09-14T22:48:49.752332+0000","last_became_peered":"2024-09-14T22:48:49.752332+0000","last_unstale":"2024-09-14T23:27:25.804380+0000","last_undegraded":"2024-09-14T23:27:25.804380+0000","last_fullsized":"2024-09-14T23:27:25.804380+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'2","last_scrub_stamp":"2024-09-14T23:27:25.804292+0000","last_deep_scrub":"464'2","last_deep_scrub_stamp":"2024-09-14T23:27:25.804292+0000","last_clean_scrub_stamp":"2024-09-14T23:27:25.804292+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T23:44:34.715026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061883999999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"464'99","reported_seq":1073,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-14T23:27:20.783099+0000","last_change":"2024-09-14T23:27:20.783182+0000","last_active":"2024-09-14T23:27:20.783099+0000","last_peered":"2024-09-14T23:27:20.783099+0000","last_clean":"2024-09-14T23:27:20.783099+0000","last_became_active":"2024-09-14T22:48:46.320719+0000","last_became_peered":"2024-09-14T22:48:46.320719+0000","last_unstale":"2024-09-14T23:27:20.783099+0000","last_undegraded":"2024-09-14T23:27:20.783099+0000","last_fullsized":"2024-09-14T23:27:20.783099+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'99","last_scrub_stamp":"2024-09-14T23:27:20.783010+0000","last_deep_scrub":"464'99","last_deep_scrub_stamp":"2024-09-14T23:27:20.783010+0000","last_clean_scrub_stamp":"2024-09-14T23:27:20.783010+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:07:47.142289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5744,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97312,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274528,"statfs":{"total":96636764160,"available":96537116672,"internally_reserved":0,"allocated":3817472,"data_stored":2042528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75228,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296612,"statfs":{"total":96636764160,"available":96559730688,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738840,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125148,"kb_used_data":3788,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246692,"statfs":{"total":96636764160,"available":96508612608,"internally_reserved":0,"allocated":3878912,"data_stored":2632915,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2872,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T23:27:40.922 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-14T23:27:41.226 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-14T23:27:41.226 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-14T23:27:41.226 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-14T23:27:41.227 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:41.266 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-14T23:27:41.266 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-14T23:27:41.266 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:41.347 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-14T23:27:41.348 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-14T23:27:41.348 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:41.392 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-14T23:27:41.392 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-14T23:27:41.392 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-14T23:27:41.392 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:41.420 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-14T23:27:41.420 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-14T23:27:41.421 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-14T23:27:41.421 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T23:27:41.473 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-14T23:27:41.473 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-14T23:27:41.473 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T23:27:41.540 INFO:teuthology.orchestra.run.smithi052.stdout:2024-09-14T22:51:40.227270+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-14T23:27:41.540 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-09-14T23:27:41.541 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T23:27:41.609 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T23:27:41.676 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[WRN\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T23:27:41.743 INFO:teuthology.orchestra.run.smithi052.stdout:2024-09-14T22:51:40.227270+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-14T23:27:41.743 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T23:27:41.743 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-14T23:27:42.074 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T23:27:42.074 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-14T23:27:42.148 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T23:27:42.149 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-14T23:27:42.204 INFO:tasks.ceph:Archiving mon data... 2024-09-14T23:27:42.204 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733/data/cluster1.mon.a.tgz 2024-09-14T23:27:42.205 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-09-14T23:27:42.220 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.Qj2cYcCJ0U 2024-09-14T23:27:42.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.Qj2cYcCJ0U 2024-09-14T23:27:42.375 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.Qj2cYcCJ0U 2024-09-14T23:27:42.460 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.Qj2cYcCJ0U is 379KB 2024-09-14T23:27:42.558 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.Qj2cYcCJ0U 2024-09-14T23:27:42.574 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-14T23:27:42.574 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T23:27:42.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T23:27:42.695 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-14T23:27:42.807 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-14T23:27:42.807 INFO:tasks.ceph:Archiving crash dumps... 2024-09-14T23:27:42.809 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733/remote/smithi052/crash 2024-09-14T23:27:42.810 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-14T23:27:42.836 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733/remote/smithi089/crash 2024-09-14T23:27:42.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-14T23:27:42.866 INFO:tasks.ceph:Compressing logs... 2024-09-14T23:27:42.866 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T23:27:42.880 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T23:27:42.902 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35940.log 2024-09-14T23:27:42.922 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36863.log 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35940.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35940.log.gz 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.36863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35518.log 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36863.log.gz 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35898.log 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37199.log 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35518.log.gz 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35769.log 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35898.log.gz 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.37199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36779.log 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37199.log.gz 2024-09-14T23:27:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35769.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.35769.log.gz 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr: -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.36779.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27629.log 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36779.log.gz 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32516.log 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph.tmp-client.admin.27629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35856.log 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27629.log.gz 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.32516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37157.log 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr: 26.0% -- replaced with /var/log/ceph/cluster1-client.admin.32516.log.gz 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32268.log 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35856.log.gz 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.37157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35220.log 2024-09-14T23:27:42.924 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37157.log.gz 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.32268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32110.log 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32268.log.gz 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35516.log 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35220.log.gz 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.32110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35352.log 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32110.log.gz 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35135.log 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35516.log.gz 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37073.log 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35352.log.gz 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35276.log 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35135.log.gz 2024-09-14T23:27:42.925 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.37073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37281.log 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37073.log.gz 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35276.log.gz 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.37281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36905.log 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37281.log.gz 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36989.log 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.36905.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36905.log.gz 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32447.log 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.36989.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36989.log.gz 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35394.log 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.32447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35644.log 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32447.log.gz 2024-09-14T23:27:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35394.log.gz 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35436.log 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35644.log.gz 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37115.log 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35436.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35436.log.gz 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr: -- /var/log/ceph/cluster1-client.admin.37031.log 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.37115.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37115.log.gz 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.37031.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37031.log.gz 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr: -- /var/log/ceph/cluster1-osd.0.log 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35643.log 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35770.log 2024-09-14T23:27:42.927 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35643.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35643.log.gz 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35730.log 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35770.log.gz 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36821.log 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35730.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35730.log.gz 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32308.log 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.36821.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.36821.log.gz -5 --verbose -- /var/log/ceph/cluster1-client.admin.35642.log 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35517.log 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35642.log: /var/log/ceph/cluster1-client.admin.32308.log: 0.0% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32308.log.gz 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35642.log.gz 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37241.log 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.35517.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32348.log 2024-09-14T23:27:42.928 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35517.log.gz 2024-09-14T23:27:42.929 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.37241.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37241.log.gz 2024-09-14T23:27:42.929 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36947.log 2024-09-14T23:27:42.929 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-14T23:27:42.929 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1-client.admin.36947.log: /var/log/ceph/cluster1-client.admin.32348.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32348.log.gz 2024-09-14T23:27:42.929 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36947.log.gz 2024-09-14T23:27:42.929 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35301.log 2024-09-14T23:27:42.929 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.35301.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.35301.log.gz 2024-09-14T23:27:42.929 INFO:teuthology.orchestra.run.smithi052.stderr: 88.1% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-14T23:27:42.944 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56141.log 2024-09-14T23:27:42.944 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63077.log 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45083.log 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65530.log 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56141.log.gz 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63077.log.gz 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.45083.log: /var/log/ceph/cluster2-client.mirror.40921.log 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45833.log 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45083.log.gz 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69008.log 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65530.log.gz 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42668.log 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53176.log 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69008.log.gz 2024-09-14T23:27:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72918.log 2024-09-14T23:27:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60301.log 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53176.log.gz 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53541.log 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr: 77.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72918.log.gz 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72675.log 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60301.log: /var/log/ceph/cluster2-client.mirror.53541.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60301.log.gz 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62643.log 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53541.log.gz 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59522.log 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72675.log: /var/log/ceph/cluster1-client.mirror.62643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60691.log 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72675.log.gz 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62643.log.gz 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62953.log 2024-09-14T23:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60691.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59522.log.gz 2024-09-14T23:27:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64640.log 2024-09-14T23:27:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75299.log 2024-09-14T23:27:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62953.log: /var/log/ceph/cluster2-client.mirror.64640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73639.log 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42668.log.gz 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60691.log.gz 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58229.log 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73639.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75299.log.gz 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65675.log 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62953.log.gz 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64640.log.gz 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55082.log 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58229.log.gz 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62715.log 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73639.log.gz 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65675.log.gz 2024-09-14T23:27:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41465.log 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55082.log.gz 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68587.log 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62715.log.gz 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41465.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39926.log 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39994.log 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68587.log.gz 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35789.log 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.42750.log 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74000.log 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35789.log.gz 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39926.log.gz 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39994.log.gz 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.0.42750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57302.log 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74000.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57061.log 2024-09-14T23:27:42.959 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74000.log.gz 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56856.log 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57302.log.gz 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.57061.log: /var/log/ceph/cluster2-client.mirror.69583.log 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64578.log 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57061.log.gz 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69583.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56856.log.gz 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69932.log 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64578.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69583.log.gz 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46762.log 2024-09-14T23:27:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58554.log 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69932.log: /var/log/ceph/cluster2-client.mirror.46762.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66591.log 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64578.log.gz 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69932.log.gz 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72778.log 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46762.log.gz 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66591.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58554.log.gz 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64186.log 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66591.log.gz 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47250.log 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72778.log: /var/log/ceph/cluster2-client.mirror.64186.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72778.log.gz 2024-09-14T23:27:42.993 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72235.log 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64186.log.gz 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37629.log 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47250.log: /var/log/ceph/cluster2-client.mirror.72235.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45103.log 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72235.log.gz 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.37629.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47250.log.gz 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37485.log 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45103.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40130.log 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.37485.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45103.log.gz 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74988.log 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72043.log 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40130.log: gzip/var/log/ceph/cluster2-client.mirror.74988.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50167.log 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72043.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74988.log.gz 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47029.log 2024-09-14T23:27:42.994 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53828.log 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72043.log.gz/var/log/ceph/cluster2-client.mirror.50167.log: 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44803.log 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47029.log.gz 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53828.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50167.log.gz 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60578.log 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55123.log 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44803.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53828.log.gz 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60578.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73048.log 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44803.log.gz 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60578.log.gz 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.55123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69364.log 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70508.log 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73048.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73048.log.gz 2024-09-14T23:27:42.995 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69364.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69364.log.gz 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67169.log 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60185.log 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70508.log: /var/log/ceph/cluster1-client.mirror.67169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47509.log 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70508.log.gz 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67169.log.gz 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72589.log 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53453.log 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.72589.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47509.log.gz 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49076.log 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72589.log.gz 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65695.log 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53453.log: /var/log/ceph/cluster2-client.mirror.49076.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60185.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48544.log 2024-09-14T23:27:42.996 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.65695.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49076.log.gz 87.3% 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55537.log 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53453.log.gz 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44545.log 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.55537.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48544.log.gz 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70339.log 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55537.log.gz 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65695.log.gz 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39483.log 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50624.log 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70339.log: /var/log/ceph/cluster1-client.mirror.39483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72541.log 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70339.log.gz 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61439.log 2024-09-14T23:27:42.997 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73141.log 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39483.log.gz/var/log/ceph/cluster1-client.mirror.61439.log: 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61439.log.gz 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63033.log 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73141.log: gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73141.log.gz 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr: /var/log/ceph/cluster2-client.mirror.61689.log 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63033.log: gzip 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55123.log.gz 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60881.log 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61689.log: 88.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56921.log 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63033.log.gz 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51469.log 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41465.log.gz 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60881.log.gz 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51448.log 2024-09-14T23:27:42.998 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44545.log.gz 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61689.log.gz 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45871.log 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65939.log 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51448.log.gz 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45871.log: gzip 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56921.log.gz 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66091.log 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65939.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51469.log.gz 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45871.log.gz 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74726.log 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74923.log 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66091.log.gz 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74020.log 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74726.log.gz 2024-09-14T23:27:42.999 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74923.log: gzip -5 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50624.log.gz 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.69399.log 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74020.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74923.log.gz 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58890.log 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74020.log.gz 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72819.log 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58890.log: gzip 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69399.log.gz 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66839.log 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72819.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58890.log.gz 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45163.log 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72819.log.gz 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.36988.log 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66839.log.gz 2024-09-14T23:27:43.000 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45163.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63236.log 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.0.36988.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45163.log.gz 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75393.log 54.2% 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37485.log.gz 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68429.log 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.75393.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63236.log.gz 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44863.log 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75393.log.gz 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43613.log 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68429.log.gz 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44863.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74627.log 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43613.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44863.log.gz 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48258.log 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60438.log 2024-09-14T23:27:43.001 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74627.log.gz 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68677.log 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58084.log 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48258.log.gz/var/log/ceph/cluster1-client.mirror.60438.log: 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69710.log 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68677.log.gz 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44092.log 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69710.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58084.log.gz 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69710.log.gz 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60438.log.gz 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71576.log 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63684.log 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44092.log: /var/log/ceph/cluster2-client.mirror.71576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61259.log 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63684.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71576.log.gz 2024-09-14T23:27:43.002 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43613.log.gz 2024-09-14T23:27:43.003 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40087.log 2024-09-14T23:27:43.003 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54009.log 2024-09-14T23:27:43.003 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40087.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63684.log.gz 2024-09-14T23:27:43.003 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61259.log.gz 2024-09-14T23:27:43.003 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75070.log 2024-09-14T23:27:43.003 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51617.log 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75070.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75070.log.gz 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54009.log.gz 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73918.log 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44092.log.gz 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67709.log 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73918.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51617.log.gz 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73697.log 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73918.log.gz 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71001.log 2024-09-14T23:27:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73164.log 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.71001.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67709.log.gz 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:gzip 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71001.log.gz 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.75914.log 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73164.log: gzip -5 --verbose -- 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73164.log.gz 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr: /var/log/ceph/cluster1-client.mirror.55981.log 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73697.log: /var/log/ceph/cluster2-client.admin.75914.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68159.log 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.75914.log.gz 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73697.log.gz 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54589.log 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55981.log: /var/log/ceph/cluster1-client.mirror.68159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59638.log 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9%/var/log/ceph/cluster2-client.mirror.54589.log: -- replaced with /var/log/ceph/cluster1-client.mirror.68159.log.gz 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65939.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66880.log 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53265.log 2024-09-14T23:27:43.061 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.66880.log: -- replaced with /var/log/ceph/cluster1-client.mirror.59638.log.gz 2024-09-14T23:27:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75956.log 2024-09-14T23:27:43.061 INFO:teuthology.orchestra.run.smithi089.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55981.log.gz 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54589.log.gz 2024-09-14T23:27:43.061 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66880.log.gz 2024-09-14T23:27:43.061 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62816.log 2024-09-14T23:27:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67889.log 2024-09-14T23:27:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.75956.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75956.log.gz 2024-09-14T23:27:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62816.log: /var/log/ceph/cluster1-client.mirror.67889.log: 89.0% 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67889.log.gz 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62816.log.gz 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62249.log 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53265.log.gz 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56660.log 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41428.log 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58827.log 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55381.log 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73598.log 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.55381.log: -- replaced with /var/log/ceph/cluster1-client.mirror.58827.log.gz 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62249.log.gz 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56660.log.gz 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60650.log 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41428.log.gz 2024-09-14T23:27:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61001.log 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9%/var/log/ceph/cluster1-client.mirror.60650.log: -- replaced with /var/log/ceph/cluster2-client.mirror.73598.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51065.log 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55381.log.gz 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.37629.log.gz 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61001.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60650.log.gz 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65054.log 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51065.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61001.log.gz 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64062.log 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51065.log.gz 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40389.log 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63979.log 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64062.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65054.log.gz 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44694.log 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64062.log.gz 2024-09-14T23:27:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57876.log 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44694.log: gzip -5 --verbose 87.7% -- /var/log/ceph/cluster2-client.mirror.37804.log -- replaced with /var/log/ceph/cluster2-client.mirror.63979.log.gz 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61572.log 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57739.log 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45529.log 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61572.log.gz 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48800.log 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57876.log.gz 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57739.log.gz 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45529.log: 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54199.log 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54568.log 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54199.log: /var/log/ceph/cluster2-client.mirror.48800.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64248.log 2024-09-14T23:27:43.064 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54199.log.gz/var/log/ceph/cluster1-client.mirror.54568.log: 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr:gzip 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54568.log.gz 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68272.log 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45529.log.gz 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64248.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48800.log.gz 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43551.log 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64248.log.gz 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68272.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44694.log.gz 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68272.log.gz 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56117.log 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36549.log 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40389.log.gz 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43551.log: /var/log/ceph/cluster1-client.mirror.56117.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64392.log 2024-09-14T23:27:43.065 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.36549.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43551.log.gz 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63125.log 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56117.log.gz 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36549.log.gz 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.47573.log 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63125.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63125.log.gz 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.2.47573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61596.log 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44416.log 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64392.log.gz 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52714.log 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61596.log.gz 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38724.log 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44416.log: /var/log/ceph/cluster1-client.mirror.52714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73065.log 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.38724.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52714.log.gz 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73747.log 2024-09-14T23:27:43.066 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53944.log 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7%/var/log/ceph/cluster1-client.mirror.73747.log: -- replaced with /var/log/ceph/cluster2-client.mirror.73065.log.gz 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.42750.log.gz 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45063.log 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73747.log.gz 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75562.log 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53944.log.gz 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42535.log 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45063.log.gz 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.75562.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75562.log.gz 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38724.log.gz 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68542.log 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74147.log 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68542.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42535.log.gz 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50150.log 2024-09-14T23:27:43.067 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68542.log.gz 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40087.log.gz 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52369.log 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74147.log: /var/log/ceph/cluster2-client.mirror.50150.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44416.log.gz 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50150.log.gz 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- /var/log/ceph/cluster2-client.mirror.75493.log 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.74147.log.gz 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68784.log 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75493.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75493.log.gz 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40363.log 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68784.log: gzip 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37804.log.gz 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68784.log.gz 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr: -- /var/log/ceph/cluster2-client.mirror.66197.log 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.40363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.72080.log 2024-09-14T23:27:43.068 INFO:teuthology.orchestra.run.smithi089.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52369.log.gz 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61239.log 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61709.log 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.0.72080.log: -- replaced with /var/log/ceph/cluster2-client.mirror.66197.log.gz 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61239.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48566.log 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40130.log.gz 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61709.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40363.log.gz 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61239.log.gz 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.72080.log.gz 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61709.log.gz 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41180.log 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48566.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56069.log 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57944.log 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54838.log 2024-09-14T23:27:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56069.log: /var/log/ceph/cluster2-client.mirror.57944.log: gzip -5 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56069.log.gz 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.47722.log 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67349.log 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57944.log.gz 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47722.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41180.log.gz 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54838.log.gz 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67867.log 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67349.log: gzip 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67349.log.gz/var/log/ceph/cluster1-client.mirror.67867.log: 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67867.log.gz 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49887.log 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70798.log 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74545.log 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46912.log 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70798.log.gz 2024-09-14T23:27:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74545.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74545.log.gz 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37587.log 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72193.log 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46912.log.gz 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.37587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64351.log 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.72193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69952.log 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72193.log.gz 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64351.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64351.log.gz 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65178.log 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.47573.log.gz 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.69952.log: -- /var/log/ceph/cluster1-client.mirror.50583.log 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69952.log.gz 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47349.log 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50583.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65178.log.gz 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:gzip 87.7% -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56808.log 2024-09-14T23:27:43.071 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.50583.log.gz 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.36988.log.gz 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50877.log 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47349.log: /var/log/ceph/cluster1-client.mirror.56808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58779.log 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47349.log.gz 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56808.log.gz/var/log/ceph/cluster1-client.mirror.50877.log: 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51270.log 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49865.log 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58779.log.gz 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51270.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51270.log.gz 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48908.log 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70278.log 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49865.log.gz 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.38814.log 2024-09-14T23:27:43.072 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.70278.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48908.log.gz 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56342.log 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70278.log.gz 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75443.log 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.2.38814.log: /var/log/ceph/cluster1-client.mirror.56342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37468.log 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75443.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56342.log.gz 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75443.log.gz 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61343.log 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50288.log 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.37468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.46030.log 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61343.log: /var/log/ceph/cluster2-client.mirror.50288.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61343.log.gz 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46156.log 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.2.46030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51951.log 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46156.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46156.log.gz 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46557.log 2024-09-14T23:27:43.073 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51951.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51951.log.gz 2024-09-14T23:27:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50007.log 2024-09-14T23:27:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46557.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50877.log.gz 2024-09-14T23:27:43.074 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46557.log.gz 2024-09-14T23:27:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69974.log 2024-09-14T23:27:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50007.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50007.log.gz 2024-09-14T23:27:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49484.log 2024-09-14T23:27:43.074 INFO:teuthology.orchestra.run.smithi089.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.46030.log.gz 2024-09-14T23:27:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69974.log: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37468.log.gz 2024-09-14T23:27:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58641.log 2024-09-14T23:27:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49484.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49484.log.gz 2024-09-14T23:27:43.075 INFO:teuthology.orchestra.run.smithi089.stderr: 81.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37587.log.gz 2024-09-14T23:27:43.081 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53848.log 2024-09-14T23:27:43.082 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58641.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58641.log.gz 2024-09-14T23:27:43.087 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61948.log 2024-09-14T23:27:43.087 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53848.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53848.log.gz 2024-09-14T23:27:43.088 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53872.log 2024-09-14T23:27:43.090 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61948.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61948.log.gz 2024-09-14T23:27:43.090 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55606.log 2024-09-14T23:27:43.090 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53872.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53872.log.gz 2024-09-14T23:27:43.091 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66549.log 2024-09-14T23:27:43.113 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.55606.log: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63274.log 2024-09-14T23:27:43.114 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66549.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66549.log.gz 2024-09-14T23:27:43.115 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55606.log.gz 2024-09-14T23:27:43.123 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64661.log 2024-09-14T23:27:43.123 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63274.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63274.log.gz 2024-09-14T23:27:43.137 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69210.log 2024-09-14T23:27:43.138 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64661.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64661.log.gz 2024-09-14T23:27:43.150 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67957.log 2024-09-14T23:27:43.150 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69210.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69210.log.gz 2024-09-14T23:27:43.163 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49319.log 2024-09-14T23:27:43.164 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67957.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67957.log.gz 2024-09-14T23:27:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65633.log 2024-09-14T23:27:43.171 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49319.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49319.log.gz 2024-09-14T23:27:43.180 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42690.log 2024-09-14T23:27:43.180 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65633.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65633.log.gz 2024-09-14T23:27:43.199 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37714.log 2024-09-14T23:27:43.199 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42690.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42690.log.gz 2024-09-14T23:27:43.207 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43146.log 2024-09-14T23:27:43.217 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.51311.log 2024-09-14T23:27:43.220 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43146.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37714.log.gz 2024-09-14T23:27:43.220 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41695.log 2024-09-14T23:27:43.227 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.2.51311.log: 93.5% 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.38814.log.gz 2024-09-14T23:27:43.235 INFO:teuthology.orchestra.run.smithi052.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-14T23:27:43.235 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43146.log.gz 2024-09-14T23:27:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51017.log 2024-09-14T23:27:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44780.log 2024-09-14T23:27:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51017.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51017.log.gz 2024-09-14T23:27:43.244 INFO:teuthology.orchestra.run.smithi052.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-14T23:27:43.259 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67259.log 2024-09-14T23:27:43.266 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44780.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41695.log.gz 2024-09-14T23:27:43.266 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44780.log.gz 2024-09-14T23:27:43.279 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37748.log 2024-09-14T23:27:43.280 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67259.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67259.log.gz 2024-09-14T23:27:43.288 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47722.log.gz 2024-09-14T23:27:43.292 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50311.log 2024-09-14T23:27:43.293 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53224.log 2024-09-14T23:27:43.294 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50311.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50311.log.gz 2024-09-14T23:27:43.294 INFO:teuthology.orchestra.run.smithi089.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37748.log.gz 2024-09-14T23:27:43.305 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.51311.log.gz 2024-09-14T23:27:43.306 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35549.log 2024-09-14T23:27:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53224.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53224.log.gz 2024-09-14T23:27:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50070.log 2024-09-14T23:27:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35549.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35549.log.gz 2024-09-14T23:27:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62908.log 2024-09-14T23:27:43.310 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50070.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50070.log.gz 2024-09-14T23:27:43.310 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52055.log 2024-09-14T23:27:43.311 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62908.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62908.log.gz 2024-09-14T23:27:43.311 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41650.log 2024-09-14T23:27:43.311 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52055.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72541.log.gz 2024-09-14T23:27:43.316 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68384.log 2024-09-14T23:27:43.316 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52055.log.gz/var/log/ceph/cluster1-client.mirror.41650.log: 2024-09-14T23:27:43.316 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69562.log 2024-09-14T23:27:43.316 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76166.log 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41650.log.gz 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69562.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68384.log.gz 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69562.log.gz 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62438.log 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.76166.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72255.log 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.76166.log.gz 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48162.log 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62438.log: /var/log/ceph/cluster2-client.mirror.72255.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62438.log.gz 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72255.log.gz 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45791.log 2024-09-14T23:27:43.317 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55916.log 2024-09-14T23:27:43.320 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48162.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48162.log.gz 2024-09-14T23:27:43.321 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51641.log 2024-09-14T23:27:43.321 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55916.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45791.log.gz 2024-09-14T23:27:43.321 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55916.log.gz 2024-09-14T23:27:43.321 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42942.log 2024-09-14T23:27:43.321 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53024.log 2024-09-14T23:27:43.321 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51641.log.gz 2024-09-14T23:27:43.321 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.45403.log 2024-09-14T23:27:43.321 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53024.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42942.log.gz 2024-09-14T23:27:43.321 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74607.log 2024-09-14T23:27:43.323 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.0.45403.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53024.log.gz 2024-09-14T23:27:43.324 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42101.log 2024-09-14T23:27:43.324 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74607.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74607.log.gz 2024-09-14T23:27:43.324 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70154.log 2024-09-14T23:27:43.324 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67552.log 2024-09-14T23:27:43.325 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.70154.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70154.log.gz 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42101.log.gz 2024-09-14T23:27:43.326 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:43.326 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75009.log 2024-09-14T23:27:43.326 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67552.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67552.log.gz 2024-09-14T23:27:43.327 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64227.log 2024-09-14T23:27:43.327 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75009.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75009.log.gz 2024-09-14T23:27:43.328 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67934.log 2024-09-14T23:27:43.328 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64227.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64227.log.gz 2024-09-14T23:27:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55513.log 2024-09-14T23:27:43.330 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67934.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67934.log.gz 2024-09-14T23:27:43.330 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62023.log 2024-09-14T23:27:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55513.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55513.log.gz 2024-09-14T23:27:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58731.log 2024-09-14T23:27:43.343 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74266.log 2024-09-14T23:27:43.344 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58731.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58731.log.gz 2024-09-14T23:27:43.345 INFO:teuthology.orchestra.run.smithi089.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62023.log.gz 2024-09-14T23:27:43.345 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45971.log 2024-09-14T23:27:43.346 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74266.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74266.log.gz 2024-09-14T23:27:43.347 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49583.log 2024-09-14T23:27:43.347 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45971.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45971.log.gz 2024-09-14T23:27:43.348 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42605.log 2024-09-14T23:27:43.370 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52031.log 2024-09-14T23:27:43.377 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42605.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42605.log.gz 2024-09-14T23:27:43.381 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68564.log 2024-09-14T23:27:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52031.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52031.log.gz 2024-09-14T23:27:43.387 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36255.log 2024-09-14T23:27:43.387 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68564.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68564.log.gz 2024-09-14T23:27:43.387 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59819.log 2024-09-14T23:27:43.387 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36255.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36255.log.gz 2024-09-14T23:27:43.387 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44244.log 2024-09-14T23:27:43.387 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59819.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59819.log.gz 2024-09-14T23:27:43.387 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72338.log 2024-09-14T23:27:43.392 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44244.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44244.log.gz 2024-09-14T23:27:43.392 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75586.log 2024-09-14T23:27:43.397 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72338.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72338.log.gz 2024-09-14T23:27:43.397 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62208.log 2024-09-14T23:27:43.397 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.75586.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75586.log.gz 2024-09-14T23:27:43.397 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50288.log.gz 2024-09-14T23:27:43.397 INFO:teuthology.orchestra.run.smithi089.stderr: -- /var/log/ceph/cluster1-client.mirror.53651.log 2024-09-14T23:27:43.397 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60905.log 2024-09-14T23:27:43.397 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62208.log.gz 2024-09-14T23:27:43.397 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62313.log 2024-09-14T23:27:43.398 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60905.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53651.log.gz 2024-09-14T23:27:43.398 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47893.log 2024-09-14T23:27:43.398 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60905.log.gz 2024-09-14T23:27:43.398 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.48693.log 2024-09-14T23:27:43.399 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47893.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62313.log.gz 2024-09-14T23:27:43.399 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65260.log 2024-09-14T23:27:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.3.48693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72839.log 2024-09-14T23:27:43.411 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65260.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65260.log.gz 2024-09-14T23:27:43.416 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49583.log.gz 2024-09-14T23:27:43.417 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53896.log 2024-09-14T23:27:43.420 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48871.log 2024-09-14T23:27:43.420 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53896.log: /var/log/ceph/cluster2-client.mirror.72839.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53896.log.gz 2024-09-14T23:27:43.420 INFO:teuthology.orchestra.run.smithi089.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72839.log.gz 2024-09-14T23:27:43.420 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43210.log 2024-09-14T23:27:43.420 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54962.log 2024-09-14T23:27:43.420 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43210.log: /var/log/ceph/cluster2-client.mirror.48871.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48871.log.gz 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.45403.log.gz 2024-09-14T23:27:43.420 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:43.420 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47489.log 2024-09-14T23:27:43.421 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54962.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54962.log.gz 2024-09-14T23:27:43.422 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65281.log 2024-09-14T23:27:43.423 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47489.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47489.log.gz 2024-09-14T23:27:43.423 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57835.log 2024-09-14T23:27:43.424 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65281.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65281.log.gz 2024-09-14T23:27:43.424 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49363.log 2024-09-14T23:27:43.426 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57835.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57835.log.gz 2024-09-14T23:27:43.426 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49201.log 2024-09-14T23:27:43.427 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49363.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49363.log.gz 2024-09-14T23:27:43.427 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46376.log 2024-09-14T23:27:43.427 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49201.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49201.log.gz 2024-09-14T23:27:43.428 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64909.log 2024-09-14T23:27:43.430 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46376.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46376.log.gz 2024-09-14T23:27:43.430 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39555.log 2024-09-14T23:27:43.430 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64909.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64909.log.gz 2024-09-14T23:27:43.432 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64950.log 2024-09-14T23:27:43.432 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48321.log 2024-09-14T23:27:43.434 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64950.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64950.log.gz 2024-09-14T23:27:43.434 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72860.log 2024-09-14T23:27:43.435 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48321.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48321.log.gz 2024-09-14T23:27:43.435 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51752.log 2024-09-14T23:27:43.436 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72860.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72860.log.gz 2024-09-14T23:27:43.437 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38680.log 2024-09-14T23:27:43.439 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51752.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51752.log.gz 2024-09-14T23:27:43.440 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39555.log.gz 2024-09-14T23:27:43.440 INFO:teuthology.orchestra.run.smithi089.stderr:gzip 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45833.log.gz 2024-09-14T23:27:43.440 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67529.log 2024-09-14T23:27:43.440 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.38680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37519.log 2024-09-14T23:27:43.443 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67529.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67529.log.gz 2024-09-14T23:27:43.443 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38680.log.gz 2024-09-14T23:27:43.443 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62995.log 2024-09-14T23:27:43.443 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.37519.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37519.log.gz 2024-09-14T23:27:43.443 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58506.log 2024-09-14T23:27:43.443 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62995.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62995.log.gz 2024-09-14T23:27:43.444 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64496.log 2024-09-14T23:27:43.445 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58506.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58506.log.gz 2024-09-14T23:27:43.445 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66218.log 2024-09-14T23:27:43.445 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64496.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64496.log.gz 2024-09-14T23:27:43.446 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.41922.log 2024-09-14T23:27:43.447 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66218.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66218.log.gz 2024-09-14T23:27:43.450 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43435.log 2024-09-14T23:27:43.450 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59289.log 2024-09-14T23:27:43.450 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43435.log: /var/log/ceph/cluster1-client.mirror.3.41922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45551.log 2024-09-14T23:27:43.450 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59289.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43435.log.gz 2024-09-14T23:27:43.450 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59289.log.gz 2024-09-14T23:27:43.450 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39102.log 2024-09-14T23:27:43.450 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54745.log 2024-09-14T23:27:43.451 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61415.log 2024-09-14T23:27:43.451 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54745.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54745.log.gz 2024-09-14T23:27:43.452 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45631.log 2024-09-14T23:27:43.452 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61415.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61415.log.gz 2024-09-14T23:27:43.453 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75050.log 2024-09-14T23:27:43.453 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45631.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45631.log.gz 2024-09-14T23:27:43.453 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74246.log 2024-09-14T23:27:43.454 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75050.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75050.log.gz 2024-09-14T23:27:43.454 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40721.log 2024-09-14T23:27:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74246.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74246.log.gz 2024-09-14T23:27:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57715.log 2024-09-14T23:27:43.456 INFO:teuthology.orchestra.run.smithi089.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.41922.log.gz 2024-09-14T23:27:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40721.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40721.log.gz 2024-09-14T23:27:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65406.log 2024-09-14T23:27:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57715.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57715.log.gz 2024-09-14T23:27:43.458 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37765.log 2024-09-14T23:27:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65406.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65406.log.gz 2024-09-14T23:27:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62549.log 2024-09-14T23:27:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53764.log 2024-09-14T23:27:43.462 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62549.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62549.log.gz 2024-09-14T23:27:43.463 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59956.log 2024-09-14T23:27:43.464 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53764.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53764.log.gz 2024-09-14T23:27:43.465 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54319.log 2024-09-14T23:27:43.467 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59956.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59956.log.gz 2024-09-14T23:27:43.467 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64847.log 2024-09-14T23:27:43.468 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54319.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54319.log.gz 2024-09-14T23:27:43.468 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45774.log 2024-09-14T23:27:43.469 INFO:teuthology.orchestra.run.smithi089.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37765.log.gz 2024-09-14T23:27:43.469 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64847.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64847.log.gz 2024-09-14T23:27:43.470 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58438.log 2024-09-14T23:27:43.470 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45774.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45774.log.gz 2024-09-14T23:27:43.471 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66508.log 2024-09-14T23:27:43.471 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58438.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58438.log.gz 2024-09-14T23:27:43.472 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56298.log 2024-09-14T23:27:43.473 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66508.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66508.log.gz 2024-09-14T23:27:43.473 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49694.log 2024-09-14T23:27:43.474 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56298.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56298.log.gz 2024-09-14T23:27:43.474 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39037.log 2024-09-14T23:27:43.488 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72610.log 2024-09-14T23:27:43.494 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39037.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39102.log.gz 2024-09-14T23:27:43.495 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49887.log.gz 2024-09-14T23:27:43.496 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67102.log 2024-09-14T23:27:43.498 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72610.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72610.log.gz 2024-09-14T23:27:43.498 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65095.log 2024-09-14T23:27:43.498 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67102.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67102.log.gz 2024-09-14T23:27:43.498 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.39037.log.gz 2024-09-14T23:27:43.499 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50535.log 2024-09-14T23:27:43.499 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65095.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65095.log.gz 2024-09-14T23:27:43.500 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.36899.log 2024-09-14T23:27:43.501 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50535.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50535.log.gz 2024-09-14T23:27:43.501 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40877.log 2024-09-14T23:27:43.502 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.2.36899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65322.log 2024-09-14T23:27:43.503 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35493.log 2024-09-14T23:27:43.505 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65322.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65322.log.gz 2024-09-14T23:27:43.507 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56477.log 2024-09-14T23:27:43.507 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35493.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35493.log.gz 2024-09-14T23:27:43.508 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44072.log 2024-09-14T23:27:43.509 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56477.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56477.log.gz 2024-09-14T23:27:43.509 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71999.log 2024-09-14T23:27:43.519 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62136.log 2024-09-14T23:27:43.520 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.71999.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.36899.log.gz 2024-09-14T23:27:43.520 INFO:teuthology.orchestra.run.smithi089.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71999.log.gz 2024-09-14T23:27:43.524 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44072.log.gz 2024-09-14T23:27:43.524 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66446.log 2024-09-14T23:27:43.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62136.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62136.log.gz 2024-09-14T23:27:43.525 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47994.log 2024-09-14T23:27:43.525 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66446.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66446.log.gz 2024-09-14T23:27:43.527 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42962.log 2024-09-14T23:27:43.527 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47994.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47994.log.gz 2024-09-14T23:27:43.527 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51814.log 2024-09-14T23:27:43.527 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42962.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42962.log.gz 2024-09-14T23:27:43.528 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46136.log 2024-09-14T23:27:43.529 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51814.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51814.log.gz 2024-09-14T23:27:43.530 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58619.log 2024-09-14T23:27:43.531 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46136.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46136.log.gz 2024-09-14T23:27:43.531 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58803.log 2024-09-14T23:27:43.533 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58619.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58619.log.gz 2024-09-14T23:27:43.534 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65799.log 2024-09-14T23:27:43.536 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58803.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58803.log.gz 2024-09-14T23:27:43.537 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44330.log 2024-09-14T23:27:43.537 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65799.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65799.log.gz 2024-09-14T23:27:43.538 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68294.log 2024-09-14T23:27:43.544 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44330.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44330.log.gz 2024-09-14T23:27:43.544 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41308.log 2024-09-14T23:27:43.544 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68294.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68294.log.gz 2024-09-14T23:27:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70566.log 2024-09-14T23:27:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71805.log 2024-09-14T23:27:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70566.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40877.log.gz 2024-09-14T23:27:43.549 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70566.log.gz 2024-09-14T23:27:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39271.log 2024-09-14T23:27:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71805.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71805.log.gz 2024-09-14T23:27:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74967.log 2024-09-14T23:27:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70921.log 2024-09-14T23:27:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.74967.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74967.log.gz 2024-09-14T23:27:43.551 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47663.log 2024-09-14T23:27:43.551 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70921.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70921.log.gz 2024-09-14T23:27:43.551 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47851.log 2024-09-14T23:27:43.551 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47663.log: 79.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47663.log.gz 2024-09-14T23:27:43.554 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42173.log 2024-09-14T23:27:43.556 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47851.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47851.log.gz 2024-09-14T23:27:43.556 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62932.log 2024-09-14T23:27:43.556 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71494.log 2024-09-14T23:27:43.558 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62932.log: 87.7% 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39271.log.gz 2024-09-14T23:27:43.558 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62932.log.gz 2024-09-14T23:27:43.558 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74689.log 2024-09-14T23:27:43.559 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71494.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71494.log.gz 2024-09-14T23:27:43.561 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42173.log.gz 2024-09-14T23:27:43.561 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56189.log 2024-09-14T23:27:43.561 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74689.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74689.log.gz 2024-09-14T23:27:43.561 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70216.log 2024-09-14T23:27:43.561 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56189.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56189.log.gz 2024-09-14T23:27:43.562 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65489.log 2024-09-14T23:27:43.563 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70216.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70216.log.gz 2024-09-14T23:27:43.563 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56318.log 2024-09-14T23:27:43.564 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65489.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65489.log.gz 2024-09-14T23:27:43.565 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45611.log 2024-09-14T23:27:43.565 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56318.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56318.log.gz 2024-09-14T23:27:43.566 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76082.log 2024-09-14T23:27:43.566 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45611.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45611.log.gz 2024-09-14T23:27:43.567 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48597.log 2024-09-14T23:27:43.568 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.76082.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76082.log.gz 2024-09-14T23:27:43.569 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51665.log 2024-09-14T23:27:43.573 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48597.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48597.log.gz 2024-09-14T23:27:43.575 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55694.log 2024-09-14T23:27:43.575 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51665.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51665.log.gz 2024-09-14T23:27:43.575 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59450.log 2024-09-14T23:27:43.580 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.55694.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55694.log.gz 2024-09-14T23:27:43.582 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36759.log 2024-09-14T23:27:43.585 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59450.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59450.log.gz 2024-09-14T23:27:43.585 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73537.log 2024-09-14T23:27:43.585 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36759.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36759.log.gz 2024-09-14T23:27:43.585 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52565.log 2024-09-14T23:27:43.588 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73537.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73537.log.gz 2024-09-14T23:27:43.589 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48199.log 2024-09-14T23:27:43.589 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52565.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52565.log.gz 2024-09-14T23:27:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41554.log 2024-09-14T23:27:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48199.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48199.log.gz 2024-09-14T23:27:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52103.log 2024-09-14T23:27:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42625.log 2024-09-14T23:27:43.593 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52103.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52103.log.gz 2024-09-14T23:27:43.593 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64165.log 2024-09-14T23:27:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42625.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41554.log.gz 2024-09-14T23:27:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73677.log 2024-09-14T23:27:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64165.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64165.log.gz 2024-09-14T23:27:43.602 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42625.log.gz 2024-09-14T23:27:43.603 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52300.log 2024-09-14T23:27:43.604 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73677.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73677.log.gz 2024-09-14T23:27:43.604 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68918.log 2024-09-14T23:27:43.605 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52300.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52300.log.gz 2024-09-14T23:27:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60626.log 2024-09-14T23:27:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68918.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68918.log.gz 2024-09-14T23:27:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66404.log 2024-09-14T23:27:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60626.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60626.log.gz 2024-09-14T23:27:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58388.log 2024-09-14T23:27:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66404.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66404.log.gz 2024-09-14T23:27:43.609 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35709.log 2024-09-14T23:27:43.610 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58388.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58388.log.gz 2024-09-14T23:27:43.611 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68786.log 2024-09-14T23:27:43.611 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35709.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35709.log.gz 2024-09-14T23:27:43.611 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66570.log 2024-09-14T23:27:43.612 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68786.log: 84.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68786.log.gz 2024-09-14T23:27:43.612 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64310.log 2024-09-14T23:27:43.613 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.48693.log.gz 2024-09-14T23:27:43.615 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66570.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66570.log.gz 2024-09-14T23:27:43.615 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68940.log 2024-09-14T23:27:43.616 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64310.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64310.log.gz 2024-09-14T23:27:43.616 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60072.log 2024-09-14T23:27:43.617 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68940.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68940.log.gz 2024-09-14T23:27:43.617 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68519.log 2024-09-14T23:27:43.618 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60072.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60072.log.gz 2024-09-14T23:27:43.618 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50246.log 2024-09-14T23:27:43.619 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68519.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68519.log.gz 2024-09-14T23:27:43.620 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65819.log 2024-09-14T23:27:43.622 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50246.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50246.log.gz 2024-09-14T23:27:43.622 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71132.log 2024-09-14T23:27:43.622 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47312.log 2024-09-14T23:27:43.622 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71132.log: /var/log/ceph/cluster2-client.mirror.65819.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71132.log.gz 2024-09-14T23:27:43.624 INFO:teuthology.orchestra.run.smithi089.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65819.log.gz 2024-09-14T23:27:43.624 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63855.log 2024-09-14T23:27:43.625 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47312.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47312.log.gz 2024-09-14T23:27:43.625 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63194.log 2024-09-14T23:27:43.627 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53337.log 2024-09-14T23:27:43.627 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63855.log.gz 2024-09-14T23:27:43.627 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.39308.log 2024-09-14T23:27:43.628 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53337.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53337.log.gz 2024-09-14T23:27:43.628 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51549.log 2024-09-14T23:27:43.629 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.1.39308.log: /var/log/ceph/cluster2-client.mirror.63194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58968.log 2024-09-14T23:27:43.631 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51549.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51549.log.gz 2024-09-14T23:27:43.631 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66362.log 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63194.log.gz 2024-09-14T23:27:43.632 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:43.632 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.58968.log: -- /var/log/ceph/cluster2-client.mirror.66156.log 2024-09-14T23:27:43.632 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66362.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58968.log.gz 2024-09-14T23:27:43.633 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66156.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66156.log.gz 2024-09-14T23:27:43.634 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66362.log.gz 2024-09-14T23:27:43.635 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56003.log 2024-09-14T23:27:43.636 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70777.log 2024-09-14T23:27:43.636 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57494.log 2024-09-14T23:27:43.637 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56003.log.gz 2024-09-14T23:27:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57494.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57494.log.gz 2024-09-14T23:27:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.70777.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70777.log.gz 2024-09-14T23:27:43.639 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43287.log 2024-09-14T23:27:43.640 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39683.log 2024-09-14T23:27:43.641 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.43324.log 2024-09-14T23:27:43.641 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39683.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43287.log.gz 2024-09-14T23:27:43.643 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68002.log 2024-09-14T23:27:43.643 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.3.43324.log: -- /var/log/ceph/cluster2-client.mirror.52011.log 2024-09-14T23:27:43.643 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68002.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68002.log.gz 2024-09-14T23:27:43.643 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47893.log.gz 2024-09-14T23:27:43.643 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56567.log 2024-09-14T23:27:43.643 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52011.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52011.log.gz 2024-09-14T23:27:43.644 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73897.log 2024-09-14T23:27:43.652 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56567.log: gzip -5 --verbose 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48566.log.gz 2024-09-14T23:27:43.653 INFO:teuthology.orchestra.run.smithi089.stderr: -- /var/log/ceph/cluster2-client.mirror.73107.log 2024-09-14T23:27:43.653 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73897.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71658.log 2024-09-14T23:27:43.653 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56567.log.gz 2024-09-14T23:27:43.653 INFO:teuthology.orchestra.run.smithi089.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73897.log.gz 2024-09-14T23:27:43.653 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51862.log 2024-09-14T23:27:43.653 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73107.log: /var/log/ceph/cluster2-client.mirror.71658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32819.log 2024-09-14T23:27:43.653 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51862.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71658.log.gz 2024-09-14T23:27:43.653 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51862.log.gz 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46440.log 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.32819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45951.log 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32819.log.gz 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46440.log: gzip 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46440.log.gz 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67822.log 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42584.log 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67822.log: /var/log/ceph/cluster2-client.mirror.45951.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67822.log.gz 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47233.log 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45951.log.gz 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42584.log: gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42584.log.gz 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr: -- /var/log/ceph/cluster1-client.mirror.43897.log 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr: 92.1%/var/log/ceph/cluster2-client.mirror.47233.log: -- replaced with /var/log/ceph/cluster2-client.mirror.1.39308.log.gz 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr: 79.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47233.log.gz 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67327.log 2024-09-14T23:27:43.654 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72379.log 2024-09-14T23:27:43.655 INFO:teuthology.orchestra.run.smithi089.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43897.log.gz/var/log/ceph/cluster1-client.mirror.67327.log: 2024-09-14T23:27:43.655 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67327.log.gz 2024-09-14T23:27:43.655 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62160.log 2024-09-14T23:27:43.661 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72379.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72379.log.gz 2024-09-14T23:27:43.661 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56523.log 2024-09-14T23:27:43.661 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62160.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62160.log.gz 2024-09-14T23:27:43.661 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74422.log 2024-09-14T23:27:43.661 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56523.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56523.log.gz 2024-09-14T23:27:43.661 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64702.log 2024-09-14T23:27:43.661 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.37018.log 2024-09-14T23:27:43.661 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64702.log: /var/log/ceph/cluster2-client.mirror.74422.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74422.log.gz 2024-09-14T23:27:43.662 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68362.log 2024-09-14T23:27:43.662 INFO:teuthology.orchestra.run.smithi089.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64702.log.gz 2024-09-14T23:27:43.662 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.3.37018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56615.log 2024-09-14T23:27:43.662 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68362.log: gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68362.log.gz 2024-09-14T23:27:43.662 INFO:teuthology.orchestra.run.smithi089.stderr: /var/log/ceph/cluster1-client.mirror.44006.log 2024-09-14T23:27:43.662 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.56615.log: /var/log/ceph/cluster2-client.mirror.53287.log 2024-09-14T23:27:43.662 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56615.log.gz 2024-09-14T23:27:43.662 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44006.log: gzip/var/log/ceph/cluster2-client.mirror.53287.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53287.log.gz 2024-09-14T23:27:43.665 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39415.log 2024-09-14T23:27:43.665 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47753.log 2024-09-14T23:27:43.666 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46338.log 2024-09-14T23:27:43.666 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47753.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47753.log.gz 2024-09-14T23:27:43.667 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62619.log 2024-09-14T23:27:43.669 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46338.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44006.log.gz 2024-09-14T23:27:43.669 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41626.log 2024-09-14T23:27:43.669 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62619.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62619.log.gz 2024-09-14T23:27:43.673 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39415.log.gz 2024-09-14T23:27:43.673 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40769.log 2024-09-14T23:27:43.675 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37400.log 2024-09-14T23:27:43.675 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70066.log 2024-09-14T23:27:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.37400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44393.log 2024-09-14T23:27:43.677 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.70066.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70066.log.gz 2024-09-14T23:27:43.677 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70608.log 2024-09-14T23:27:43.680 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44393.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.43324.log.gz 2024-09-14T23:27:43.682 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70608.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41626.log.gz 2024-09-14T23:27:43.682 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51222.log 2024-09-14T23:27:43.682 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40769.log.gz 2024-09-14T23:27:43.683 INFO:teuthology.orchestra.run.smithi089.stderr: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37400.log.gz 2024-09-14T23:27:43.688 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44393.log.gz 2024-09-14T23:27:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47573.log 2024-09-14T23:27:43.692 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51222.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51222.log.gz 2024-09-14T23:27:43.692 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46457.log 2024-09-14T23:27:43.702 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.47573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55585.log 2024-09-14T23:27:43.703 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46457.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46457.log.gz 2024-09-14T23:27:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66921.log 2024-09-14T23:27:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55585.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55585.log.gz 2024-09-14T23:27:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66921.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66921.log.gz 2024-09-14T23:27:43.707 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66715.log 2024-09-14T23:27:43.707 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75111.log 2024-09-14T23:27:43.708 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66715.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66715.log.gz 2024-09-14T23:27:43.711 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60671.log 2024-09-14T23:27:43.712 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75111.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75111.log.gz 2024-09-14T23:27:43.716 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75788.log 2024-09-14T23:27:43.718 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60671.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60671.log.gz 2024-09-14T23:27:43.722 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42398.log 2024-09-14T23:27:43.722 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.75788.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75788.log.gz 2024-09-14T23:27:43.723 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69648.log 2024-09-14T23:27:43.727 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42398.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42398.log.gz 2024-09-14T23:27:43.727 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.71716.log 2024-09-14T23:27:43.732 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69648.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69648.log.gz 2024-09-14T23:27:43.732 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35625.log 2024-09-14T23:27:43.740 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.3.71716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64682.log 2024-09-14T23:27:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47531.log 2024-09-14T23:27:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64682.log: /var/log/ceph/cluster2-client.admin.35625.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35625.log.gz 2024-09-14T23:27:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58060.log 2024-09-14T23:27:43.760 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47531.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64682.log.gz 2024-09-14T23:27:43.765 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44029.log 2024-09-14T23:27:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58060.log: 87.8% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43210.log.gz 2024-09-14T23:27:43.770 INFO:teuthology.orchestra.run.smithi089.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.71716.log.gz 2024-09-14T23:27:43.771 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58060.log.gz 2024-09-14T23:27:43.771 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51593.log 2024-09-14T23:27:43.771 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64268.log 2024-09-14T23:27:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51593.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51593.log.gz 2024-09-14T23:27:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49845.log 2024-09-14T23:27:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64268.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64268.log.gz 2024-09-14T23:27:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36675.log 2024-09-14T23:27:43.775 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49845.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49845.log.gz 2024-09-14T23:27:43.776 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74401.log 2024-09-14T23:27:43.777 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36675.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36675.log.gz 2024-09-14T23:27:43.777 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71408.log 2024-09-14T23:27:43.777 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.74401.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74401.log.gz 2024-09-14T23:27:43.778 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54428.log 2024-09-14T23:27:43.779 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71408.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71408.log.gz 2024-09-14T23:27:43.779 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69853.log 2024-09-14T23:27:43.781 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54428.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54428.log.gz 2024-09-14T23:27:43.781 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37697.log 2024-09-14T23:27:43.782 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69853.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69853.log.gz 2024-09-14T23:27:43.782 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53044.log 2024-09-14T23:27:43.783 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58012.log 2024-09-14T23:27:43.791 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53044.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53044.log.gz 2024-09-14T23:27:43.791 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52127.log 2024-09-14T23:27:43.791 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44029.log.gz 2024-09-14T23:27:43.791 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58012.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58012.log.gz 2024-09-14T23:27:43.791 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51246.log 2024-09-14T23:27:43.791 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52127.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52127.log.gz 2024-09-14T23:27:43.791 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62778.log 2024-09-14T23:27:43.792 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51246.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51246.log.gz 2024-09-14T23:27:43.792 INFO:teuthology.orchestra.run.smithi089.stderr: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37697.log.gz 2024-09-14T23:27:43.792 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55465.log 2024-09-14T23:27:43.792 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62778.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62778.log.gz 2024-09-14T23:27:43.803 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63896.log 2024-09-14T23:27:43.804 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.55465.log: 86.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55465.log.gz 2024-09-14T23:27:43.805 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51089.log 2024-09-14T23:27:43.810 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63896.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63896.log.gz 2024-09-14T23:27:43.810 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65427.log 2024-09-14T23:27:43.815 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51089.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51089.log.gz 2024-09-14T23:27:43.820 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69915.log 2024-09-14T23:27:43.823 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65427.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65427.log.gz 2024-09-14T23:27:43.824 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44651.log 2024-09-14T23:27:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69915.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69915.log.gz 2024-09-14T23:27:43.829 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27744.log 2024-09-14T23:27:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39531.log 2024-09-14T23:27:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.tmp-client.admin.27744.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69974.log.gz 2024-09-14T23:27:43.843 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27744.log.gz 2024-09-14T23:27:43.843 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44651.log.gz 2024-09-14T23:27:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63917.log 2024-09-14T23:27:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51973.log 2024-09-14T23:27:43.844 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63917.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63917.log.gz 2024-09-14T23:27:43.845 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74748.log 2024-09-14T23:27:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51973.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51973.log.gz 2024-09-14T23:27:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64103.log 2024-09-14T23:27:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64888.log 2024-09-14T23:27:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64103.log: /var/log/ceph/cluster2-client.mirror.74748.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64103.log.gz 2024-09-14T23:27:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69255.log 2024-09-14T23:27:43.850 INFO:teuthology.orchestra.run.smithi052.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-14T23:27:43.852 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64888.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39531.log.gz 2024-09-14T23:27:43.853 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64888.log.gz 2024-09-14T23:27:43.859 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75538.log 2024-09-14T23:27:43.859 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7%/var/log/ceph/cluster1-client.mirror.69255.log: -- replaced with /var/log/ceph/cluster2-client.mirror.45551.log.gz 2024-09-14T23:27:43.859 INFO:teuthology.orchestra.run.smithi089.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69255.log.gz 2024-09-14T23:27:43.859 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69771.log 2024-09-14T23:27:43.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.75538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74668.log 2024-09-14T23:27:43.859 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75538.log.gz 2024-09-14T23:27:43.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69771.log: gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69771.log.gz 2024-09-14T23:27:43.859 INFO:teuthology.orchestra.run.smithi089.stderr: /var/log/ceph/cluster2-client.mirror.47292.log 2024-09-14T23:27:43.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60602.log 2024-09-14T23:27:43.860 INFO:teuthology.orchestra.run.smithi089.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74668.log.gz 2024-09-14T23:27:43.862 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63663.log 2024-09-14T23:27:43.866 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60602.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60602.log.gz 2024-09-14T23:27:43.867 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71976.log 2024-09-14T23:27:43.872 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63663.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63663.log.gz 2024-09-14T23:27:43.872 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70819.log 2024-09-14T23:27:43.876 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.71976.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71976.log.gz 2024-09-14T23:27:43.876 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72318.log 2024-09-14T23:27:43.880 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70819.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70819.log.gz 2024-09-14T23:27:43.880 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37612.log 2024-09-14T23:27:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72318.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72318.log.gz 2024-09-14T23:27:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54496.log 2024-09-14T23:27:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.37612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.36856.log 2024-09-14T23:27:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54496.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54496.log.gz 2024-09-14T23:27:43.883 INFO:teuthology.orchestra.run.smithi089.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.37612.log.gz 2024-09-14T23:27:43.887 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49928.log 2024-09-14T23:27:43.887 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.1.36856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75830.log 2024-09-14T23:27:43.887 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49928.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.37018.log.gz 2024-09-14T23:27:43.887 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49928.log.gz 2024-09-14T23:27:43.887 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47573.log.gz 2024-09-14T23:27:43.887 INFO:teuthology.orchestra.run.smithi089.stderr: -- /var/log/ceph/cluster2-client.mirror.45704.log 2024-09-14T23:27:43.887 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.75830.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75830.log.gz 2024-09-14T23:27:43.887 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52148.log 2024-09-14T23:27:43.888 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45704.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45704.log.gz 2024-09-14T23:27:43.888 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67574.log 2024-09-14T23:27:43.890 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52148.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52148.log.gz 2024-09-14T23:27:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48011.log 2024-09-14T23:27:43.892 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67574.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67574.log.gz 2024-09-14T23:27:43.892 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62486.log 2024-09-14T23:27:43.893 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48011.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48011.log.gz 2024-09-14T23:27:43.894 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35574.log 2024-09-14T23:27:43.898 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62486.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62486.log.gz 2024-09-14T23:27:43.900 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47871.log 2024-09-14T23:27:43.900 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42560.log 2024-09-14T23:27:43.900 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47871.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47871.log.gz 2024-09-14T23:27:43.901 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47072.log 2024-09-14T23:27:43.903 INFO:teuthology.orchestra.run.smithi089.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.admin.35574.log.gz 2024-09-14T23:27:43.905 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42560.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42560.log.gz 2024-09-14T23:27:43.905 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42874.log 2024-09-14T23:27:43.905 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47072.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47072.log.gz 2024-09-14T23:27:43.906 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66989.log 2024-09-14T23:27:43.910 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.42874.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42874.log.gz 2024-09-14T23:27:43.915 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43527.log 2024-09-14T23:27:43.915 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.66989.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66989.log.gz 2024-09-14T23:27:43.915 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68609.log 2024-09-14T23:27:43.918 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43527.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43527.log.gz 2024-09-14T23:27:43.918 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66947.log 2024-09-14T23:27:43.918 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68609.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68609.log.gz 2024-09-14T23:27:43.919 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44588.log 2024-09-14T23:27:43.919 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.66947.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66947.log.gz 2024-09-14T23:27:43.919 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56455.log 2024-09-14T23:27:43.928 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44588.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44588.log.gz 2024-09-14T23:27:43.928 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72080.log 2024-09-14T23:27:43.928 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74208.log 2024-09-14T23:27:43.932 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.72080.log: 75.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72080.log.gz 2024-09-14T23:27:43.934 INFO:teuthology.orchestra.run.smithi089.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56455.log.gz 2024-09-14T23:27:43.934 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43765.log 2024-09-14T23:27:43.936 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74208.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74208.log.gz 2024-09-14T23:27:43.941 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64599.log 2024-09-14T23:27:43.950 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32576.log 2024-09-14T23:27:43.955 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64599.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43765.log.gz 2024-09-14T23:27:43.955 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64599.log.gz 2024-09-14T23:27:43.955 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63751.log 2024-09-14T23:27:43.960 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.32576.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32576.log.gz 2024-09-14T23:27:43.962 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67012.log 2024-09-14T23:27:43.962 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63751.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63751.log.gz 2024-09-14T23:27:43.962 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53200.log 2024-09-14T23:27:43.962 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67012.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67012.log.gz 2024-09-14T23:27:43.964 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51689.log 2024-09-14T23:27:43.964 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53200.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53200.log.gz 2024-09-14T23:27:43.967 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46537.log 2024-09-14T23:27:43.969 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51689.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51689.log.gz 2024-09-14T23:27:43.970 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55626.log 2024-09-14T23:27:43.971 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46537.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46537.log.gz 2024-09-14T23:27:43.972 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66280.log 2024-09-14T23:27:43.979 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55626.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55626.log.gz 2024-09-14T23:27:43.980 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63318.log 2024-09-14T23:27:43.980 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66280.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66280.log.gz 2024-09-14T23:27:43.981 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41602.log 2024-09-14T23:27:43.982 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63318.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63318.log.gz 2024-09-14T23:27:43.982 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45324.log 2024-09-14T23:27:43.985 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41602.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41602.log.gz 2024-09-14T23:27:43.986 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69323.log 2024-09-14T23:27:43.986 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45324.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45324.log.gz 2024-09-14T23:27:43.988 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.65939.log 2024-09-14T23:27:43.988 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69323.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69323.log.gz 2024-09-14T23:27:43.988 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59683.log 2024-09-14T23:27:44.008 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.0.65939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39061.log 2024-09-14T23:27:44.017 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59683.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32656.log 2024-09-14T23:27:44.022 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39061.log: -- replaced with /var/log/ceph/cluster2-client.mirror.59683.log.gz 2024-09-14T23:27:44.030 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61281.log 2024-09-14T23:27:44.030 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.32656.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39061.log.gz 2024-09-14T23:27:44.030 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32656.log.gz 2024-09-14T23:27:44.039 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.65939.log.gz 2024-09-14T23:27:44.039 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63730.log 2024-09-14T23:27:44.039 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61281.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61281.log.gz 2024-09-14T23:27:44.041 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58868.log 2024-09-14T23:27:44.041 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63730.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63730.log.gz 2024-09-14T23:27:44.041 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42986.log 2024-09-14T23:27:44.042 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58868.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58868.log.gz 2024-09-14T23:27:44.043 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-14T23:27:44.050 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42986.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42986.log.gz 2024-09-14T23:27:44.051 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51931.log 2024-09-14T23:27:44.056 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-osd.2.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47531.log.gz 2024-09-14T23:27:44.057 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54295.log 2024-09-14T23:27:44.058 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51931.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51931.log.gz 2024-09-14T23:27:44.060 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57856.log 2024-09-14T23:27:44.060 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61813.log 2024-09-14T23:27:44.060 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54295.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54295.log.gz 2024-09-14T23:27:44.061 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75190.log 2024-09-14T23:27:44.061 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61813.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61813.log.gz 2024-09-14T23:27:44.062 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49911.log 2024-09-14T23:27:44.062 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75190.log: 82.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75190.log.gz 2024-09-14T23:27:44.064 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50107.log 2024-09-14T23:27:44.064 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49911.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49911.log.gz 2024-09-14T23:27:44.065 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56252.log 2024-09-14T23:27:44.069 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50107.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50107.log.gz 2024-09-14T23:27:44.069 INFO:teuthology.orchestra.run.smithi089.stderr:gzip 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70608.log.gz 2024-09-14T23:27:44.069 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57856.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74229.log 2024-09-14T23:27:44.069 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69053.log 2024-09-14T23:27:44.069 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56252.log: /var/log/ceph/cluster2-client.mirror.74229.log: 79.7% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56252.log.gz 2024-09-14T23:27:44.069 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.74229.log.gz 2024-09-14T23:27:44.070 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57856.log.gz 2024-09-14T23:27:44.070 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.37434.log 2024-09-14T23:27:44.070 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69053.log: gzip -5 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69053.log.gz 2024-09-14T23:27:44.070 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.69874.log 2024-09-14T23:27:44.070 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.37434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51489.log 2024-09-14T23:27:44.070 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69874.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69874.log.gz 2024-09-14T23:27:44.071 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.49583.log 2024-09-14T23:27:44.072 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53627.log 2024-09-14T23:27:44.072 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.1.49583.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51489.log.gz 2024-09-14T23:27:44.076 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36423.log 2024-09-14T23:27:44.076 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53627.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53627.log.gz 2024-09-14T23:27:44.076 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43921.log 2024-09-14T23:27:44.076 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.36423.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36423.log.gz 2024-09-14T23:27:44.076 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.71001.log 2024-09-14T23:27:44.076 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43921.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43921.log.gz 2024-09-14T23:27:44.076 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73210.log 2024-09-14T23:27:44.076 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.1.71001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44565.log 2024-09-14T23:27:44.078 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73210.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73210.log.gz 2024-09-14T23:27:44.078 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50788.log 2024-09-14T23:27:44.081 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44565.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37434.log.gz 2024-09-14T23:27:44.082 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.1.71001.log.gz 2024-09-14T23:27:44.082 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65840.log 2024-09-14T23:27:44.082 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50788.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50788.log.gz 2024-09-14T23:27:44.084 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56680.log 2024-09-14T23:27:44.084 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65840.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65840.log.gz 2024-09-14T23:27:44.084 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49136.log 2024-09-14T23:27:44.085 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56680.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44565.log.gz 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56680.log.gz 2024-09-14T23:27:44.085 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:44.085 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63567.log 2024-09-14T23:27:44.087 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49136.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49136.log.gz 2024-09-14T23:27:44.087 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66259.log 2024-09-14T23:27:44.088 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63567.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63567.log.gz 2024-09-14T23:27:44.088 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50764.log 2024-09-14T23:27:44.092 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66259.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56760.log 2024-09-14T23:27:44.092 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6%/var/log/ceph/cluster1-client.mirror.50764.log: -- replaced with /var/log/ceph/cluster2-client.mirror.66259.log.gz 2024-09-14T23:27:44.092 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64620.log 2024-09-14T23:27:44.092 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.50764.log.gz 2024-09-14T23:27:44.093 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70528.log 2024-09-14T23:27:44.093 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56760.log: /var/log/ceph/cluster2-client.mirror.64620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71825.log 2024-09-14T23:27:44.093 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70528.log: 87.8% 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56760.log.gz 2024-09-14T23:27:44.093 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64620.log.gz 2024-09-14T23:27:44.097 INFO:teuthology.orchestra.run.smithi089.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70528.log.gz 2024-09-14T23:27:44.097 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62112.log 2024-09-14T23:27:44.097 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49282.log 2024-09-14T23:27:44.097 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62112.log: /var/log/ceph/cluster2-client.mirror.71825.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62112.log.gz 2024-09-14T23:27:44.097 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58301.log 2024-09-14T23:27:44.097 INFO:teuthology.orchestra.run.smithi089.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71825.log.gz 2024-09-14T23:27:44.098 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56901.log 2024-09-14T23:27:44.098 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58301.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58301.log.gz 2024-09-14T23:27:44.098 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64516.log 2024-09-14T23:27:44.100 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56901.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56901.log.gz 2024-09-14T23:27:44.100 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49828.log 2024-09-14T23:27:44.102 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64516.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64516.log.gz 2024-09-14T23:27:44.103 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51838.log 2024-09-14T23:27:44.103 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49828.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49828.log.gz 2024-09-14T23:27:44.104 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39729.log 2024-09-14T23:27:44.105 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49282.log: /var/log/ceph/cluster1-client.mirror.51838.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51838.log.gz 2024-09-14T23:27:44.105 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.57126.log 2024-09-14T23:27:44.106 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.39729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44674.log 2024-09-14T23:27:44.110 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.0.57126.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49282.log.gz 2024-09-14T23:27:44.111 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69030.log 2024-09-14T23:27:44.121 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44674.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44674.log.gz 2024-09-14T23:27:44.121 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64930.log 2024-09-14T23:27:44.126 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69030.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.57126.log.gz 2024-09-14T23:27:44.126 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69030.log.gz 2024-09-14T23:27:44.126 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43614.log 2024-09-14T23:27:44.126 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73369.log 2024-09-14T23:27:44.127 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43614.log: /var/log/ceph/cluster2-client.mirror.64930.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64930.log.gz 2024-09-14T23:27:44.127 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49422.log 2024-09-14T23:27:44.127 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73369.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73369.log.gz 2024-09-14T23:27:44.128 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66487.log 2024-09-14T23:27:44.129 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49422.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43614.log.gz 2024-09-14T23:27:44.131 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66487.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66487.log.gz 2024-09-14T23:27:44.134 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48053.log 2024-09-14T23:27:44.134 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54769.log 2024-09-14T23:27:44.137 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48053.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74748.log.gz 2024-09-14T23:27:44.137 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69669.log 2024-09-14T23:27:44.138 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54769.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54769.log.gz 2024-09-14T23:27:44.138 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56784.log 2024-09-14T23:27:44.140 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69669.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69669.log.gz 2024-09-14T23:27:44.140 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50397.log 2024-09-14T23:27:44.140 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56784.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56784.log.gz 2024-09-14T23:27:44.140 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41360.log 2024-09-14T23:27:44.142 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50397.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50397.log.gz 2024-09-14T23:27:44.142 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47051.log 2024-09-14T23:27:44.143 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.41360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73284.log 2024-09-14T23:27:44.146 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47051.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41360.log.gz 2024-09-14T23:27:44.149 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46338.log.gz 2024-09-14T23:27:44.149 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65033.log 2024-09-14T23:27:44.149 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73284.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73284.log.gz 2024-09-14T23:27:44.149 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68963.log 2024-09-14T23:27:44.150 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65033.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65033.log.gz 2024-09-14T23:27:44.150 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74706.log 2024-09-14T23:27:44.151 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68963.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68963.log.gz 2024-09-14T23:27:44.153 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72277.log 2024-09-14T23:27:44.153 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74706.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74706.log.gz 2024-09-14T23:27:44.153 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54520.log 2024-09-14T23:27:44.157 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72277.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72277.log.gz 2024-09-14T23:27:44.158 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39124.log 2024-09-14T23:27:44.158 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54520.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54520.log.gz 2024-09-14T23:27:44.163 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74586.log 2024-09-14T23:27:44.163 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39124.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.39729.log.gz 2024-09-14T23:27:44.164 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69751.log 2024-09-14T23:27:44.164 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39124.log.gz 2024-09-14T23:27:44.167 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74586.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74586.log.gz 2024-09-14T23:27:44.167 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44158.log 2024-09-14T23:27:44.167 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69751.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69751.log.gz 2024-09-14T23:27:44.167 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60530.log 2024-09-14T23:27:44.176 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44158.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49694.log.gz 2024-09-14T23:27:44.177 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47410.log 2024-09-14T23:27:44.177 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44158.log.gz 2024-09-14T23:27:44.177 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60530.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60530.log.gz 2024-09-14T23:27:44.183 INFO:teuthology.orchestra.run.smithi089.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.49583.log.gz 2024-09-14T23:27:44.184 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72880.log 2024-09-14T23:27:44.184 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68407.log 2024-09-14T23:27:44.186 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64330.log 2024-09-14T23:27:44.186 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47410.log.gz 2024-09-14T23:27:44.186 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68407.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68407.log.gz 2024-09-14T23:27:44.186 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72880.log.gz 2024-09-14T23:27:44.186 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50899.log 2024-09-14T23:27:44.187 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64330.log: /var/log/ceph/cluster2-client.mirror.50899.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50899.log.gz 2024-09-14T23:27:44.187 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64330.log.gz 2024-09-14T23:27:44.187 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69833.log 2024-09-14T23:27:44.189 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73027.log 2024-09-14T23:27:44.189 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75152.log 2024-09-14T23:27:44.189 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69833.log.gz 2024-09-14T23:27:44.190 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73027.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73027.log.gz 2024-09-14T23:27:44.190 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39596.log 2024-09-14T23:27:44.194 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.75152.log: -- /var/log/ceph/cluster2-client.mirror.48820.log 2024-09-14T23:27:44.194 INFO:teuthology.orchestra.run.smithi089.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75152.log.gz 2024-09-14T23:27:44.194 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49178.log 2024-09-14T23:27:44.194 INFO:teuthology.orchestra.run.smithi089.stderr:gzip/var/log/ceph/cluster2-client.mirror.48820.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.48417.log 2024-09-14T23:27:44.194 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52690.log 2024-09-14T23:27:44.194 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48820.log.gz 2024-09-14T23:27:44.194 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.3.48417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51910.log 2024-09-14T23:27:44.194 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52690.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52690.log.gz 2024-09-14T23:27:44.194 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48969.log 2024-09-14T23:27:44.195 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51910.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51910.log.gz 2024-09-14T23:27:44.195 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40972.log 2024-09-14T23:27:44.196 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48969.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48969.log.gz 2024-09-14T23:27:44.197 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37546.log 2024-09-14T23:27:44.198 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.40972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53128.log 2024-09-14T23:27:44.198 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.37546.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37546.log.gz 2024-09-14T23:27:44.199 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.40972.log.gz 2024-09-14T23:27:44.206 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39970.log 2024-09-14T23:27:44.209 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53128.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53128.log.gz 2024-09-14T23:27:44.210 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43103.log 2024-09-14T23:27:44.210 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64475.log 2024-09-14T23:27:44.215 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43103.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39970.log.gz 2024-09-14T23:27:44.218 INFO:teuthology.orchestra.run.smithi089.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.48417.log.gz 2024-09-14T23:27:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70196.log 2024-09-14T23:27:44.221 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64475.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64475.log.gz 2024-09-14T23:27:44.221 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60953.log 2024-09-14T23:27:44.221 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70196.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70196.log.gz 2024-09-14T23:27:44.222 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69419.log 2024-09-14T23:27:44.222 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60953.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43103.log.gz 2024-09-14T23:27:44.222 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60953.log.gz 2024-09-14T23:27:44.224 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68137.log 2024-09-14T23:27:44.226 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61218.log 2024-09-14T23:27:44.226 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68137.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69419.log.gz 2024-09-14T23:27:44.226 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68137.log.gz 2024-09-14T23:27:44.226 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72566.log 2024-09-14T23:27:44.226 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46296.log 2024-09-14T23:27:44.228 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.72566.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61218.log.gz 2024-09-14T23:27:44.228 INFO:teuthology.orchestra.run.smithi089.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72566.log.gz 2024-09-14T23:27:44.228 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41244.log 2024-09-14T23:27:44.232 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46296.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46296.log.gz 2024-09-14T23:27:44.232 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55803.log 2024-09-14T23:27:44.232 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50716.log 2024-09-14T23:27:44.232 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55803.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55803.log.gz 2024-09-14T23:27:44.232 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55489.log 2024-09-14T23:27:44.232 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44178.log 2024-09-14T23:27:44.232 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50716.log.gz 2024-09-14T23:27:44.232 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45366.log 2024-09-14T23:27:44.236 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44178.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55489.log.gz 2024-09-14T23:27:44.236 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65881.log 2024-09-14T23:27:44.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45366.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39596.log.gz 2024-09-14T23:27:44.241 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44178.log.gz 2024-09-14T23:27:44.241 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49039.log 2024-09-14T23:27:44.244 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65881.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65881.log.gz 2024-09-14T23:27:44.244 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39013.log 2024-09-14T23:27:44.244 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49039.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49039.log.gz 2024-09-14T23:27:44.245 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50087.log 2024-09-14T23:27:44.245 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48358.log 2024-09-14T23:27:44.246 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50087.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50087.log.gz 2024-09-14T23:27:44.247 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43943.log 2024-09-14T23:27:44.247 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48358.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48358.log.gz 2024-09-14T23:27:44.248 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64082.log 2024-09-14T23:27:44.254 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43943.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43943.log.gz 2024-09-14T23:27:44.255 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74378.log 2024-09-14T23:27:44.257 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64082.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64082.log.gz 2024-09-14T23:27:44.257 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49056.log 2024-09-14T23:27:44.258 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.74378.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74378.log.gz 2024-09-14T23:27:44.258 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65198.log 2024-09-14T23:27:44.260 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49056.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49056.log.gz 2024-09-14T23:27:44.260 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72716.log 2024-09-14T23:27:44.262 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65198.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65198.log.gz 2024-09-14T23:27:44.262 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69812.log 2024-09-14T23:27:44.263 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72716.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72716.log.gz 2024-09-14T23:27:44.263 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46745.log 2024-09-14T23:27:44.265 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69812.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69812.log.gz 2024-09-14T23:27:44.265 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46199.log 2024-09-14T23:27:44.267 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46745.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46745.log.gz 2024-09-14T23:27:44.267 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60349.log 2024-09-14T23:27:44.267 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46199.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46199.log.gz 2024-09-14T23:27:44.268 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64806.log 2024-09-14T23:27:44.269 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60349.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60349.log.gz 2024-09-14T23:27:44.269 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39082.log 2024-09-14T23:27:44.270 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64806.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64806.log.gz 2024-09-14T23:27:44.270 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40042.log 2024-09-14T23:27:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39082.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66342.log 2024-09-14T23:27:44.273 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58707.log 2024-09-14T23:27:44.276 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66342.log: 87.9% 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40042.log.gz 2024-09-14T23:27:44.276 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66342.log.gz 2024-09-14T23:27:44.277 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67124.log 2024-09-14T23:27:44.278 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58707.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58707.log.gz 2024-09-14T23:27:44.282 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49218.log 2024-09-14T23:27:44.282 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.39013.log.gz 2024-09-14T23:27:44.283 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67124.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67124.log.gz 2024-09-14T23:27:44.283 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65136.log 2024-09-14T23:27:44.285 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49218.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49218.log.gz 2024-09-14T23:27:44.285 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48221.log 2024-09-14T23:27:44.286 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65136.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73107.log.gz 2024-09-14T23:27:44.286 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73328.log 2024-09-14T23:27:44.286 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48221.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65136.log.gz 2024-09-14T23:27:44.296 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51511.log 2024-09-14T23:27:44.296 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44502.log 2024-09-14T23:27:44.297 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73328.log: /var/log/ceph/cluster2-client.mirror.51511.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73328.log.gz 2024-09-14T23:27:44.297 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51511.log.gz 2024-09-14T23:27:44.297 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53108.log 2024-09-14T23:27:44.308 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44502.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44502.log.gz 2024-09-14T23:27:44.308 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41040.log 2024-09-14T23:27:44.309 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41244.log.gz 2024-09-14T23:27:44.311 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53108.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53108.log.gz 2024-09-14T23:27:44.311 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65861.log 2024-09-14T23:27:44.313 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38989.log 2024-09-14T23:27:44.313 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65861.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65861.log.gz 2024-09-14T23:27:44.313 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44923.log 2024-09-14T23:27:44.313 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.38989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38658.log 2024-09-14T23:27:44.315 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44923.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44923.log.gz 2024-09-14T23:27:44.316 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59385.log 2024-09-14T23:27:44.316 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.38658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60418.log 2024-09-14T23:27:44.316 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59385.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59385.log.gz 2024-09-14T23:27:44.320 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% 87.7% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38989.log.gz 2024-09-14T23:27:44.321 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.38658.log.gz 2024-09-14T23:27:44.321 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41040.log.gz 2024-09-14T23:27:44.322 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41738.log 2024-09-14T23:27:44.324 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60418.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60418.log.gz 2024-09-14T23:27:44.325 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45993.log 2024-09-14T23:27:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70712.log 2024-09-14T23:27:44.343 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74785.log 2024-09-14T23:27:44.343 INFO:teuthology.orchestra.run.smithi089.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39082.log.gz 2024-09-14T23:27:44.343 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.70712.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70712.log.gz 2024-09-14T23:27:44.355 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62333.log 2024-09-14T23:27:44.355 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.74785.log: 81.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74785.log.gz 2024-09-14T23:27:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75132.log 2024-09-14T23:27:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62333.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62333.log.gz 2024-09-14T23:27:44.382 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71090.log 2024-09-14T23:27:44.382 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75132.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75132.log.gz 2024-09-14T23:27:44.397 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41738.log.gz 2024-09-14T23:27:44.400 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73959.log 2024-09-14T23:27:44.400 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47292.log.gz 2024-09-14T23:27:44.400 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56740.log 2024-09-14T23:27:44.400 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.73959.log: -- replaced with /var/log/ceph/cluster2-client.mirror.71090.log.gz 2024-09-14T23:27:44.400 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73959.log.gz 2024-09-14T23:27:44.400 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70319.log 2024-09-14T23:27:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56740.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56740.log.gz 2024-09-14T23:27:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41859.log 2024-09-14T23:27:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70319.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70319.log.gz 2024-09-14T23:27:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67304.log 2024-09-14T23:27:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.41859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51886.log 2024-09-14T23:27:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67304.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67304.log.gz 2024-09-14T23:27:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71210.log 2024-09-14T23:27:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51886.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41859.log.gz 2024-09-14T23:27:44.405 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51886.log.gz 2024-09-14T23:27:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68339.log 2024-09-14T23:27:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71210.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71210.log.gz 2024-09-14T23:27:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68632.log 2024-09-14T23:27:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68339.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68339.log.gz 2024-09-14T23:27:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36129.log 2024-09-14T23:27:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68632.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68632.log.gz 2024-09-14T23:27:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69120.log 2024-09-14T23:27:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36129.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36129.log.gz 2024-09-14T23:27:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55292.log 2024-09-14T23:27:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69120.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69120.log.gz 2024-09-14T23:27:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67282.log 2024-09-14T23:27:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55292.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55292.log.gz 2024-09-14T23:27:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.72918.log 2024-09-14T23:27:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67282.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67282.log.gz 2024-09-14T23:27:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52955.log 2024-09-14T23:27:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.1.72918.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.72918.log.gz 2024-09-14T23:27:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48142.log 2024-09-14T23:27:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52955.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52955.log.gz 2024-09-14T23:27:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51130.log 2024-09-14T23:27:44.428 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62836.log 2024-09-14T23:27:44.430 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51130.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51130.log.gz 2024-09-14T23:27:44.441 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40245.log 2024-09-14T23:27:44.441 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62836.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62836.log.gz 2024-09-14T23:27:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63958.log 2024-09-14T23:27:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40245.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47051.log.gz 2024-09-14T23:27:44.461 INFO:teuthology.orchestra.run.smithi089.stderr: 73.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40245.log.gz 2024-09-14T23:27:44.466 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40172.log 2024-09-14T23:27:44.469 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63958.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63958.log.gz 2024-09-14T23:27:44.469 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68204.log 2024-09-14T23:27:44.470 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62507.log 2024-09-14T23:27:44.470 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68204.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68204.log.gz 2024-09-14T23:27:44.478 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40172.log.gz 2024-09-14T23:27:44.478 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54986.log 2024-09-14T23:27:44.487 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48300.log 2024-09-14T23:27:44.496 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54986.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62357.log 2024-09-14T23:27:44.500 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48300.log: -- replaced with /var/log/ceph/cluster1-client.mirror.54986.log.gz 2024-09-14T23:27:44.501 INFO:teuthology.orchestra.run.smithi089.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62507.log.gz 2024-09-14T23:27:44.505 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52541.log 2024-09-14T23:27:44.505 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62357.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62357.log.gz 2024-09-14T23:27:44.514 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64289.log 2024-09-14T23:27:44.527 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52541.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52541.log.gz 2024-09-14T23:27:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59867.log 2024-09-14T23:27:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64289.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64289.log.gz 2024-09-14T23:27:44.560 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66859.log 2024-09-14T23:27:44.561 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59867.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59867.log.gz 2024-09-14T23:27:44.581 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54918.log 2024-09-14T23:27:44.583 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66859.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66859.log.gz 2024-09-14T23:27:44.594 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36840.log 2024-09-14T23:27:44.595 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54918.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54918.log.gz 2024-09-14T23:27:44.609 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43722.log 2024-09-14T23:27:44.613 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.36840.log: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.36856.log.gz 2024-09-14T23:27:44.613 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52887.log 2024-09-14T23:27:44.622 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43722.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43722.log.gz 2024-09-14T23:27:44.623 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37325.log 2024-09-14T23:27:44.627 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52887.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52887.log.gz 2024-09-14T23:27:44.631 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36063.log 2024-09-14T23:27:44.631 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.37325.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37325.log.gz 2024-09-14T23:27:44.631 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-14T23:27:44.631 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36063.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36063.log.gz 2024-09-14T23:27:44.631 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46870.log 2024-09-14T23:27:44.631 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66300.log 2024-09-14T23:27:44.632 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46870.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46870.log.gz 2024-09-14T23:27:44.632 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45811.log 2024-09-14T23:27:44.632 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66300.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66300.log.gz 2024-09-14T23:27:44.636 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40921.log.gz 2024-09-14T23:27:44.639 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69300.log 2024-09-14T23:27:44.639 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45811.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45811.log.gz 2024-09-14T23:27:44.639 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44903.log 2024-09-14T23:27:44.639 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69300.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69300.log.gz 2024-09-14T23:27:44.640 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61789.log 2024-09-14T23:27:44.642 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44903.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44903.log.gz 2024-09-14T23:27:44.643 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68069.log 2024-09-14T23:27:44.643 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61789.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61789.log.gz 2024-09-14T23:27:44.643 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44823.log 2024-09-14T23:27:44.643 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68069.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68069.log.gz 2024-09-14T23:27:44.643 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59566.log 2024-09-14T23:27:44.646 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44823.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44823.log.gz 2024-09-14T23:27:44.646 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61064.log 2024-09-14T23:27:44.647 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59566.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59566.log.gz 2024-09-14T23:27:44.647 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54631.log 2024-09-14T23:27:44.647 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61064.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61064.log.gz 2024-09-14T23:27:44.647 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72737.log 2024-09-14T23:27:44.648 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54631.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54631.log.gz 2024-09-14T23:27:44.648 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.40245.log 2024-09-14T23:27:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72737.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72737.log.gz 2024-09-14T23:27:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66777.log 2024-09-14T23:27:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.1.40245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63366.log 2024-09-14T23:27:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66777.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66777.log.gz 2024-09-14T23:27:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54472.log 2024-09-14T23:27:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63366.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63366.log.gz 2024-09-14T23:27:44.671 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43234.log 2024-09-14T23:27:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54472.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64537.log 2024-09-14T23:27:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.43234.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.43234.log.gz 2024-09-14T23:27:44.684 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54472.log.gz 2024-09-14T23:27:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70298.log 2024-09-14T23:27:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64537.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64537.log.gz 2024-09-14T23:27:44.711 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54544.log 2024-09-14T23:27:44.712 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70298.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70298.log.gz 2024-09-14T23:27:44.722 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53003.log 2024-09-14T23:27:44.723 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54544.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54544.log.gz 2024-09-14T23:27:44.731 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74463.log 2024-09-14T23:27:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53003.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49422.log.gz 2024-09-14T23:27:44.744 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53003.log.gz 2024-09-14T23:27:44.745 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48142.log.gz 2024-09-14T23:27:44.745 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63435.log 2024-09-14T23:27:44.745 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54448.log 2024-09-14T23:27:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63435.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63435.log.gz 2024-09-14T23:27:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62691.log 2024-09-14T23:27:44.750 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54448.log: 87.4% 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74463.log.gz 2024-09-14T23:27:44.750 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54448.log.gz 2024-09-14T23:27:44.750 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60554.log 2024-09-14T23:27:44.750 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47950.log 2024-09-14T23:27:44.751 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53744.log 2024-09-14T23:27:44.751 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47972.log 2024-09-14T23:27:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59725.log 2024-09-14T23:27:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60554.log: /var/log/ceph/cluster2-client.mirror.47950.log: 87.5% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60554.log.gz 2024-09-14T23:27:44.754 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47950.log.gz 2024-09-14T23:27:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62691.log: /var/log/ceph/cluster2-client.mirror.53744.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62691.log.gz 2024-09-14T23:27:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70942.log 2024-09-14T23:27:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59725.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53744.log.gz 2024-09-14T23:27:44.755 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59725.log.gz 2024-09-14T23:27:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74188.log 2024-09-14T23:27:44.756 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70942.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70942.log.gz 2024-09-14T23:27:44.756 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44843.log 2024-09-14T23:27:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74188.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74188.log.gz 2024-09-14T23:27:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50646.log 2024-09-14T23:27:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44843.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44843.log.gz 2024-09-14T23:27:44.763 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37018.log 2024-09-14T23:27:44.768 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50646.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50646.log.gz 2024-09-14T23:27:44.773 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64434.log 2024-09-14T23:27:44.773 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45366.log.gz 2024-09-14T23:27:44.773 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64785.log 2024-09-14T23:27:44.774 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64434.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64434.log.gz/var/log/ceph/cluster1-client.mirror.37018.log: 2024-09-14T23:27:44.775 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46216.log 2024-09-14T23:27:44.775 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64785.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64785.log.gz 2024-09-14T23:27:44.776 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69604.log 2024-09-14T23:27:44.779 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46216.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46216.log.gz 2024-09-14T23:27:44.779 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54721.log 2024-09-14T23:27:44.779 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36171.log 2024-09-14T23:27:44.779 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54721.log: /var/log/ceph/cluster1-client.mirror.69604.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69604.log.gz 2024-09-14T23:27:44.779 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54721.log.gz 2024-09-14T23:27:44.779 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.58968.log 2024-09-14T23:27:44.782 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45854.log 2024-09-14T23:27:44.782 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.0.58968.log: /var/log/ceph/cluster2-client.admin.36171.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36171.log.gz 2024-09-14T23:27:44.782 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42712.log 2024-09-14T23:27:44.782 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45854.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45854.log.gz 2024-09-14T23:27:44.782 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55805.log 2024-09-14T23:27:44.786 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42712.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48221.log.gz 2024-09-14T23:27:44.789 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49178.log.gz 2024-09-14T23:27:44.790 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51366.log 2024-09-14T23:27:44.792 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55805.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55805.log.gz 2024-09-14T23:27:44.792 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47131.log 2024-09-14T23:27:44.793 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57611.log 2024-09-14T23:27:44.794 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47131.log: /var/log/ceph/cluster1-client.mirror.51366.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51366.log.gz 2024-09-14T23:27:44.795 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59542.log 2024-09-14T23:27:44.797 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57611.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57611.log.gz 2024-09-14T23:27:44.797 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45259.log 2024-09-14T23:27:44.797 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59542.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59542.log.gz 2024-09-14T23:27:44.798 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40814.log 2024-09-14T23:27:44.804 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.45259.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.58968.log.gz 2024-09-14T23:27:44.805 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45259.log.gz 2024-09-14T23:27:44.806 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67057.log 2024-09-14T23:27:44.806 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58578.log 2024-09-14T23:27:44.808 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68699.log 2024-09-14T23:27:44.808 INFO:teuthology.orchestra.run.smithi089.stderr: 87.0%/var/log/ceph/cluster1-client.mirror.58578.log: -- replaced with /var/log/ceph/cluster1-client.mirror.67057.log.gz 2024-09-14T23:27:44.808 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48507.log 2024-09-14T23:27:44.808 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68699.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68699.log.gz 2024-09-14T23:27:44.808 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58578.log.gz 2024-09-14T23:27:44.812 INFO:teuthology.orchestra.run.smithi089.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.40245.log.gz 2024-09-14T23:27:44.815 INFO:teuthology.orchestra.run.smithi089.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37018.log.gz 2024-09-14T23:27:44.815 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70860.log 2024-09-14T23:27:44.816 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32616.log 2024-09-14T23:27:44.817 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70860.log: 87.7% 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40814.log.gz 2024-09-14T23:27:44.817 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70860.log.gz 2024-09-14T23:27:44.817 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48507.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48507.log.gz 2024-09-14T23:27:44.817 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74336.log 2024-09-14T23:27:44.819 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.32616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47770.log 2024-09-14T23:27:44.819 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32616.log.gz 2024-09-14T23:27:44.819 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.74336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42149.log 2024-09-14T23:27:44.819 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74336.log.gz 2024-09-14T23:27:44.819 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47770.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47770.log.gz 2024-09-14T23:27:44.820 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39245.log 2024-09-14T23:27:44.820 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43575.log 2024-09-14T23:27:44.822 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.39245.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39245.log.gz 2024-09-14T23:27:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47109.log 2024-09-14T23:27:44.825 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70754.log 2024-09-14T23:27:44.825 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47109.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47109.log.gz 2024-09-14T23:27:44.826 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43575.log.gz 2024-09-14T23:27:44.828 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45993.log.gz 2024-09-14T23:27:44.828 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68317.log 2024-09-14T23:27:44.832 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.70754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46359.log 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr: 85.0%/var/log/ceph/cluster1-client.mirror.68317.log: -- replaced with /var/log/ceph/cluster1-client.mirror.70754.log.gz 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr:gzip 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68317.log.gz 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75279.log 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75704.log 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46359.log.gz 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53920.log 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75279.log.gz/var/log/ceph/cluster2-client.admin.75704.log: 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75704.log.gz 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51041.log 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60277.log 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53920.log.gz 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54677.log 2024-09-14T23:27:44.833 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51041.log.gz 2024-09-14T23:27:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60277.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60277.log.gz 2024-09-14T23:27:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58185.log 2024-09-14T23:27:44.836 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54677.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54677.log.gz 2024-09-14T23:27:44.836 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52628.log 2024-09-14T23:27:44.836 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39146.log 2024-09-14T23:27:44.836 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58185.log: /var/log/ceph/cluster2-client.mirror.52628.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58185.log.gz 2024-09-14T23:27:44.836 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52628.log.gz 2024-09-14T23:27:44.836 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70689.log 2024-09-14T23:27:44.837 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41386.log 2024-09-14T23:27:44.837 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59703.log 2024-09-14T23:27:44.840 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.70689.log: /var/log/ceph/cluster2-client.mirror.41386.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70689.log.gz 2024-09-14T23:27:44.840 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62667.log 2024-09-14T23:27:44.840 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57013.log 2024-09-14T23:27:44.840 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65074.log 2024-09-14T23:27:44.840 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62667.log.gz 2024-09-14T23:27:44.840 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47790.log 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57013.log.gz 2024-09-14T23:27:44.840 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:44.840 INFO:teuthology.orchestra.run.smithi089.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59703.log.gz 2024-09-14T23:27:44.853 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65074.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65074.log.gz 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37663.log 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56390.log 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37663.log: /var/log/ceph/cluster2-client.mirror.47790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71953.log 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56390.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47790.log.gz 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37301.log 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56390.log.gz 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.71953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37824.log 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62599.log 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37824.log: /var/log/ceph/cluster1-client.admin.37301.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37301.log.gz 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50836.log 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57988.log 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62599.log.gz 2024-09-14T23:27:44.854 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50836.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71953.log.gz 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50836.log.gz 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37824.log.gz 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44983.log 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr: 54.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37663.log.gz 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48989.log 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44983.log: /var/log/ceph/cluster1-client.mirror.57988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54271.log 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48989.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44983.log.gz 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57988.log.gz 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48989.log.gz 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56210.log 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45143.log 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54271.log.gz 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68024.log 2024-09-14T23:27:44.855 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45143.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45143.log.gz 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41885.log 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56210.log.gz 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68024.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68024.log.gz 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41758.log 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47930.log 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50993.log 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47930.log: 88.6% 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41885.log.gz 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47930.log.gz 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60397.log 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr: 94.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64124.log 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36840.log.gz/var/log/ceph/cluster1-client.mirror.50993.log: 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60397.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50993.log.gz 2024-09-14T23:27:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73719.log 2024-09-14T23:27:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39756.log 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73719.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64124.log.gz 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60397.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39507.log 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.39756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60755.log 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48053.log.gz 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64558.log 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59843.log 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60755.log: /var/log/ceph/cluster2-client.mirror.64558.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60755.log.gz 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64558.log.gz 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60028.log 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56435.log 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60028.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59843.log.gz 2024-09-14T23:27:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44350.log 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47153.log 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44350.log: /var/log/ceph/cluster2-client.mirror.56435.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56435.log.gz 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60028.log.gz 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39756.log.gz 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53989.log 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39507.log.gz 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47153.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47153.log.gz 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52411.log 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53989.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44350.log.gz 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42077.log 2024-09-14T23:27:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52411.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53989.log.gz 2024-09-14T23:27:44.873 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48614.log 2024-09-14T23:27:44.873 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52411.log.gz 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70467.log 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43479.log 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70467.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48614.log.gz 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70467.log.gz 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52807.log 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46119.log 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52807.log: /var/log/ceph/cluster1-client.mirror.43479.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43479.log.gz 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50266.log 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46119.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52807.log.gz 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46119.log.gz 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70089.log 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68474.log 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50266.log: /var/log/ceph/cluster1-client.mirror.70089.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50266.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.49739.log 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:44.874 INFO:teuthology.orchestra.run.smithi089.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70089.log.gz 2024-09-14T23:27:44.875 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54223.log 2024-09-14T23:27:44.875 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68474.log: /var/log/ceph/cluster2-client.mirror.2.49739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39639.log 2024-09-14T23:27:44.875 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68474.log.gz 2024-09-14T23:27:44.875 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46992.log 2024-09-14T23:27:44.875 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39639.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54223.log.gz 2024-09-14T23:27:44.877 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43058.log 2024-09-14T23:27:44.877 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46992.log: gzip -5 --verbose 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46992.log.gz 2024-09-14T23:27:44.877 INFO:teuthology.orchestra.run.smithi089.stderr: -- /var/log/ceph/cluster1-client.mirror.55738.log 2024-09-14T23:27:44.878 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43058.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43058.log.gz 2024-09-14T23:27:44.878 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38702.log 2024-09-14T23:27:44.879 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55738.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55738.log.gz 2024-09-14T23:27:44.879 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52849.log 2024-09-14T23:27:44.881 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.38702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50049.log 2024-09-14T23:27:44.881 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47270.log 2024-09-14T23:27:44.881 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50049.log: /var/log/ceph/cluster2-client.mirror.52849.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52849.log.gz 2024-09-14T23:27:44.882 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63342.log 2024-09-14T23:27:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47270.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.49739.log.gz 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47270.log.gz 2024-09-14T23:27:44.884 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53675.log 2024-09-14T23:27:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63342.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63342.log.gz 2024-09-14T23:27:44.884 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38702.log.gz 2024-09-14T23:27:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72297.log 2024-09-14T23:27:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53675.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53675.log.gz 2024-09-14T23:27:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44135.log 2024-09-14T23:27:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72297.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66901.log 2024-09-14T23:27:44.887 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72297.log.gz 2024-09-14T23:27:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56943.log 2024-09-14T23:27:44.888 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66901.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66901.log.gz 2024-09-14T23:27:44.888 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38777.log 2024-09-14T23:27:44.889 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56943.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56943.log.gz 2024-09-14T23:27:44.890 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67034.log 2024-09-14T23:27:44.890 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44135.log: /var/log/ceph/cluster2-client.mirror.38777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43034.log 2024-09-14T23:27:44.891 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38777.log.gz 2024-09-14T23:27:44.892 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67034.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67034.log.gz 2024-09-14T23:27:44.892 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41386.log.gz 2024-09-14T23:27:44.894 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44135.log.gz 2024-09-14T23:27:44.903 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56993.log 2024-09-14T23:27:44.903 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62973.log 2024-09-14T23:27:44.903 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43034.log.gz 2024-09-14T23:27:44.903 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72696.log 2024-09-14T23:27:44.903 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62973.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56993.log.gz 2024-09-14T23:27:44.903 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61644.log 2024-09-14T23:27:44.905 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72696.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62973.log.gz 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72696.log.gz 2024-09-14T23:27:44.905 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36003.log 2024-09-14T23:27:44.905 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61644.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61644.log.gz 2024-09-14T23:27:44.907 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71637.log 2024-09-14T23:27:44.907 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36003.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36003.log.gz 2024-09-14T23:27:44.907 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:44.907 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42237.log 2024-09-14T23:27:44.908 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49011.log 2024-09-14T23:27:44.909 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42237.log: /var/log/ceph/cluster2-client.mirror.71637.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71637.log.gz 2024-09-14T23:27:44.909 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38814.log 2024-09-14T23:27:44.909 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70131.log 2024-09-14T23:27:44.913 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49011.log: /var/log/ceph/cluster2-client.mirror.38814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47329.log 2024-09-14T23:27:44.929 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64372.log 2024-09-14T23:27:44.930 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47329.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47329.log.gz 2024-09-14T23:27:44.934 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.70131.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70131.log.gz 2024-09-14T23:27:44.947 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56880.log 2024-09-14T23:27:44.948 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64372.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64372.log.gz 2024-09-14T23:27:44.954 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68744.log 2024-09-14T23:27:44.963 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37731.log 2024-09-14T23:27:44.967 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68744.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56880.log.gz 2024-09-14T23:27:44.968 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42237.log.gz 2024-09-14T23:27:44.973 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38941.log 2024-09-14T23:27:44.976 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37731.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68744.log.gz 2024-09-14T23:27:44.978 INFO:teuthology.orchestra.run.smithi089.stderr: 57.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37731.log.gz 2024-09-14T23:27:44.990 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63477.log 2024-09-14T23:27:44.991 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.38941.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38941.log.gz 2024-09-14T23:27:45.003 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69441.log 2024-09-14T23:27:45.004 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63477.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63477.log.gz 2024-09-14T23:27:45.012 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42345.log 2024-09-14T23:27:45.021 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41060.log 2024-09-14T23:27:45.034 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.42345.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42345.log.gz 2024-09-14T23:27:45.036 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48300.log.gz 2024-09-14T23:27:45.041 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35877.log 2024-09-14T23:27:45.043 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41060.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41060.log.gz 2024-09-14T23:27:45.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73856.log 2024-09-14T23:27:45.059 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35877.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35877.log.gz 2024-09-14T23:27:45.086 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49529.log 2024-09-14T23:27:45.086 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73856.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73856.log.gz 2024-09-14T23:27:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65571.log 2024-09-14T23:27:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49529.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49529.log.gz 2024-09-14T23:27:45.105 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.46616.log 2024-09-14T23:27:45.117 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57763.log 2024-09-14T23:27:45.118 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.1.46616.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38814.log.gz 2024-09-14T23:27:45.119 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.36951.log 2024-09-14T23:27:45.119 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65571.log.gz 2024-09-14T23:27:45.122 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57763.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57763.log.gz 2024-09-14T23:27:45.122 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44608.log 2024-09-14T23:27:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.3.36951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71514.log 2024-09-14T23:27:45.145 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68092.log 2024-09-14T23:27:45.146 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71514.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71514.log.gz 2024-09-14T23:27:45.149 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44608.log.gz 2024-09-14T23:27:45.153 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73619.log 2024-09-14T23:27:45.153 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68092.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68092.log.gz 2024-09-14T23:27:45.163 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54817.log 2024-09-14T23:27:45.163 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73619.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73619.log.gz 2024-09-14T23:27:45.165 INFO:teuthology.orchestra.run.smithi089.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.46616.log.gz 2024-09-14T23:27:45.165 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43787.log 2024-09-14T23:27:45.166 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54817.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54817.log.gz 2024-09-14T23:27:45.167 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67507.log 2024-09-14T23:27:45.168 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43787.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43787.log.gz 2024-09-14T23:27:45.168 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70839.log 2024-09-14T23:27:45.170 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67507.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67507.log.gz 2024-09-14T23:27:45.170 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58325.log 2024-09-14T23:27:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70839.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70839.log.gz 2024-09-14T23:27:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40107.log 2024-09-14T23:27:45.173 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58325.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58325.log.gz 2024-09-14T23:27:45.173 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61480.log 2024-09-14T23:27:45.174 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74313.log 2024-09-14T23:27:45.175 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61480.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42077.log.gz 2024-09-14T23:27:45.175 INFO:teuthology.orchestra.run.smithi089.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61480.log.gz 2024-09-14T23:27:45.177 INFO:teuthology.orchestra.run.smithi089.stderr: 93.1% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-14T23:27:45.178 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42149.log.gz 2024-09-14T23:27:45.179 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.36951.log.gz 2024-09-14T23:27:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.74313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44717.log 2024-09-14T23:27:45.181 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74313.log.gz 2024-09-14T23:27:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64454.log 2024-09-14T23:27:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44717.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44717.log.gz 2024-09-14T23:27:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55940.log 2024-09-14T23:27:45.193 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64454.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64454.log.gz 2024-09-14T23:27:45.193 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76124.log 2024-09-14T23:27:45.193 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39639.log.gz 2024-09-14T23:27:45.195 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55940.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55940.log.gz 2024-09-14T23:27:45.195 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37015.log 2024-09-14T23:27:45.195 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.76124.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76124.log.gz 2024-09-14T23:27:45.195 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56639.log 2024-09-14T23:27:45.196 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.37015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44963.log 2024-09-14T23:27:45.197 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56639.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56639.log.gz 2024-09-14T23:27:45.197 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69143.log 2024-09-14T23:27:45.199 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44963.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37015.log.gz 2024-09-14T23:27:45.199 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44963.log.gz 2024-09-14T23:27:45.199 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38485.log 2024-09-14T23:27:45.200 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69143.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69143.log.gz 2024-09-14T23:27:45.200 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61837.log 2024-09-14T23:27:45.201 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.38485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63834.log 2024-09-14T23:27:45.202 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61837.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61837.log.gz 2024-09-14T23:27:45.210 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56702.log 2024-09-14T23:27:45.213 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63834.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63834.log.gz 2024-09-14T23:27:45.213 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47472.log 2024-09-14T23:27:45.213 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56702.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56702.log.gz 2024-09-14T23:27:45.225 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32755.log 2024-09-14T23:27:45.225 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47472.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47472.log.gz 2024-09-14T23:27:45.225 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44115.log 2024-09-14T23:27:45.226 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.32755.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32755.log.gz 2024-09-14T23:27:45.233 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.37587.log 2024-09-14T23:27:45.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44115.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44115.log.gz 2024-09-14T23:27:45.239 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59936.log 2024-09-14T23:27:45.240 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.1.37587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70962.log 2024-09-14T23:27:45.242 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59936.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59936.log.gz 2024-09-14T23:27:45.246 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74648.log 2024-09-14T23:27:45.247 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70962.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70962.log.gz 2024-09-14T23:27:45.247 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59269.log 2024-09-14T23:27:45.248 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74648.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74648.log.gz 2024-09-14T23:27:45.249 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66673.log 2024-09-14T23:27:45.250 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59269.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59269.log.gz 2024-09-14T23:27:45.251 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66673.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.37587.log.gz 2024-09-14T23:27:45.251 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54360.log 2024-09-14T23:27:45.252 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66673.log.gz 2024-09-14T23:27:45.254 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54360.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54360.log.gz 2024-09-14T23:27:45.260 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58366.log 2024-09-14T23:27:45.260 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74357.log 2024-09-14T23:27:45.268 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58366.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58366.log.gz 2024-09-14T23:27:45.269 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42750.log 2024-09-14T23:27:45.269 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74357.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74357.log.gz 2024-09-14T23:27:45.273 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66611.log 2024-09-14T23:27:45.287 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42750.log: 95.4%/var/log/ceph/cluster2-client.mirror.66611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53519.log 2024-09-14T23:27:45.287 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42750.log.gz 2024-09-14T23:27:45.288 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66611.log.gz 2024-09-14T23:27:45.290 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45236.log 2024-09-14T23:27:45.295 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53519.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53519.log.gz 2024-09-14T23:27:45.296 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45403.log 2024-09-14T23:27:45.301 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-14T23:27:45.312 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61460.log 2024-09-14T23:27:45.319 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75872.log 2024-09-14T23:27:45.333 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61460.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61460.log.gz 2024-09-14T23:27:45.343 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73448.log 2024-09-14T23:27:45.343 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.75872.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75872.log.gz 2024-09-14T23:27:45.352 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60257.log 2024-09-14T23:27:45.352 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73448.log: 83.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73448.log.gz 2024-09-14T23:27:45.355 INFO:teuthology.orchestra.run.smithi089.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40107.log.gz 2024-09-14T23:27:45.370 INFO:teuthology.orchestra.run.smithi089.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45236.log.gz 2024-09-14T23:27:45.375 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56093.log 2024-09-14T23:27:45.376 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60257.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60257.log.gz 2024-09-14T23:27:45.384 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69098.log 2024-09-14T23:27:45.388 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56093.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41308.log.gz 2024-09-14T23:27:45.389 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56093.log.gz 2024-09-14T23:27:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75418.log 2024-09-14T23:27:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69098.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69098.log.gz 2024-09-14T23:27:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49506.log 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45403.log.gz 2024-09-14T23:27:45.416 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.75418.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75418.log.gz 2024-09-14T23:27:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75468.log 2024-09-14T23:27:45.416 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47131.log.gz 2024-09-14T23:27:45.418 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41922.log 2024-09-14T23:27:45.419 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75468.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75468.log.gz 2024-09-14T23:27:45.419 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70110.log 2024-09-14T23:27:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44943.log 2024-09-14T23:27:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70110.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70110.log.gz 2024-09-14T23:27:45.421 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42125.log 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44943.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44943.log.gz 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65240.log 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66529.log 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65240.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65240.log.gz 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35315.log 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66529.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66529.log.gz 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55058.log 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41758.log.gz 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35315.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35315.log.gz 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58205.log 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55058.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55058.log.gz 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr: 75.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41922.log.gz 2024-09-14T23:27:45.434 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42125.log.gz 2024-09-14T23:27:45.436 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57695.log 2024-09-14T23:27:45.437 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58205.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58205.log.gz 2024-09-14T23:27:45.437 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67394.log 2024-09-14T23:27:45.437 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57695.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57695.log.gz 2024-09-14T23:27:45.437 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49400.log 2024-09-14T23:27:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67394.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67394.log.gz 2024-09-14T23:27:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59472.log 2024-09-14T23:27:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49400.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49400.log.gz 2024-09-14T23:27:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67912.log 2024-09-14T23:27:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59472.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50049.log.gz 2024-09-14T23:27:45.440 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59472.log.gz 2024-09-14T23:27:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59978.log 2024-09-14T23:27:45.442 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67912.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67912.log.gz 2024-09-14T23:27:45.442 INFO:teuthology.orchestra.run.smithi089.stderr: /var/log/ceph/cluster2-client.mirror.63814.log 2024-09-14T23:27:45.442 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59978.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59978.log.gz 2024-09-14T23:27:45.443 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48120.log 2024-09-14T23:27:45.444 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63814.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63814.log.gz 2024-09-14T23:27:45.444 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47972.log.gz 2024-09-14T23:27:45.444 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72420.log 2024-09-14T23:27:45.444 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.48120.log: --verbose -- /var/log/ceph/cluster2-client.mirror.45043.log 2024-09-14T23:27:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72420.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48120.log.gz 2024-09-14T23:27:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71232.log 2024-09-14T23:27:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45043.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72420.log.gz 2024-09-14T23:27:45.446 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43324.log 2024-09-14T23:27:45.446 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45043.log.gz 2024-09-14T23:27:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.71232.log: --verbose -- /var/log/ceph/cluster2-client.mirror.1.55805.log 2024-09-14T23:27:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40485.log 2024-09-14T23:27:45.457 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.1.55805.log: gzip -5 --verbose 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69441.log.gz 2024-09-14T23:27:45.457 INFO:teuthology.orchestra.run.smithi089.stderr: -- /var/log/ceph/cluster2-client.mirror.39782.log 2024-09-14T23:27:45.457 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48656.log 2024-09-14T23:27:45.457 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40485.log: /var/log/ceph/cluster2-client.mirror.39782.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75091.log 2024-09-14T23:27:45.464 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48656.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39782.log.gz 2024-09-14T23:27:45.466 INFO:teuthology.orchestra.run.smithi089.stderr: 94.2% 81.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40485.log.gz 2024-09-14T23:27:45.466 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.55805.log.gz 2024-09-14T23:27:45.468 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46949.log 2024-09-14T23:27:45.469 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75091.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75091.log.gz 2024-09-14T23:27:45.470 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44631.log 2024-09-14T23:27:45.470 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46949.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46949.log.gz 2024-09-14T23:27:45.471 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59361.log 2024-09-14T23:27:45.475 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44631.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43324.log.gz 2024-09-14T23:27:45.476 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44631.log.gz 2024-09-14T23:27:45.477 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57631.log 2024-09-14T23:27:45.478 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59361.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59361.log.gz 2024-09-14T23:27:45.478 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.60792.log 2024-09-14T23:27:45.479 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41286.log 2024-09-14T23:27:45.481 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57631.log: /var/log/ceph/cluster2-client.mirror.1.60792.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57631.log.gz 2024-09-14T23:27:45.481 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41802.log 2024-09-14T23:27:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42487.log 2024-09-14T23:27:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41802.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42712.log.gz 2024-09-14T23:27:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.70378.log 2024-09-14T23:27:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42487.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41286.log.gz 2024-09-14T23:27:45.484 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42487.log.gz 2024-09-14T23:27:45.492 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46499.log 2024-09-14T23:27:45.493 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.2.70378.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73719.log.gz 2024-09-14T23:27:45.494 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57518.log 2024-09-14T23:27:45.495 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52931.log 2024-09-14T23:27:45.496 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57518.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57518.log.gz 2024-09-14T23:27:45.496 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61170.log 2024-09-14T23:27:45.496 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52931.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52931.log.gz 2024-09-14T23:27:45.497 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43745.log 2024-09-14T23:27:45.500 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61170.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61170.log.gz 2024-09-14T23:27:45.501 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36381.log 2024-09-14T23:27:45.501 INFO:teuthology.orchestra.run.smithi089.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.70378.log.gz 2024-09-14T23:27:45.501 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46833.log 2024-09-14T23:27:45.501 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.36381.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36381.log.gz 2024-09-14T23:27:45.501 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60048.log 2024-09-14T23:27:45.501 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46833.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46833.log.gz 2024-09-14T23:27:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42463.log 2024-09-14T23:27:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60048.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60048.log.gz 2024-09-14T23:27:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57566.log 2024-09-14T23:27:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42463.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42463.log.gz 2024-09-14T23:27:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63053.log 2024-09-14T23:27:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57566.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57566.log.gz 2024-09-14T23:27:45.505 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43745.log.gz 2024-09-14T23:27:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58530.log 2024-09-14T23:27:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63053.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.60792.log.gz 2024-09-14T23:27:45.508 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63053.log.gz 2024-09-14T23:27:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58346.log 2024-09-14T23:27:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43986.log 2024-09-14T23:27:45.508 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58530.log.gz 2024-09-14T23:27:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48241.log 2024-09-14T23:27:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58036.log 2024-09-14T23:27:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48241.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48241.log.gz 2024-09-14T23:27:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41221.log 2024-09-14T23:27:45.511 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58346.log.gz 2024-09-14T23:27:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50696.log 2024-09-14T23:27:45.512 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58036.log.gz 2024-09-14T23:27:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41221.log: /var/log/ceph/cluster2-client.mirror.43986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76206.log 2024-09-14T23:27:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50696.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50696.log.gz 2024-09-14T23:27:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.73448.log 2024-09-14T23:27:45.514 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.76206.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76206.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39308.log 2024-09-14T23:27:45.514 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:45.515 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.0.73448.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.73448.log.gz 2024-09-14T23:27:45.516 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46396.log 2024-09-14T23:27:45.516 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66424.log 2024-09-14T23:27:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39308.log: /var/log/ceph/cluster2-client.mirror.46396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69075.log 2024-09-14T23:27:45.524 INFO:teuthology.orchestra.run.smithi089.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43986.log.gz 2024-09-14T23:27:45.524 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46396.log.gz/var/log/ceph/cluster2-client.mirror.66424.log: 2024-09-14T23:27:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64041.log 2024-09-14T23:27:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69075.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37680.log 2024-09-14T23:27:45.524 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69075.log.gz 2024-09-14T23:27:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64041.log: 73.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39308.log.gz 2024-09-14T23:27:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53603.log 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37680.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64041.log.gz 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74944.log 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66424.log.gz 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58482.log 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.74944.log: -- replaced with /var/log/ceph/cluster2-client.mirror.53603.log.gz 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65592.log 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74944.log.gz 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37680.log.gz 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43455.log 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58482.log: /var/log/ceph/cluster2-client.mirror.65592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48338.log 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43455.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58482.log.gz 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65592.log.gz 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66383.log 2024-09-14T23:27:45.525 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43455.log.gz 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36988.log 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52389.log 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.36988.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48338.log.gz/var/log/ceph/cluster2-client.mirror.66383.log: 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65902.log 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52389.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66383.log.gz 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66735.log 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72170.log 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65902.log: /var/log/ceph/cluster2-client.mirror.66735.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65902.log.gz 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr: 80.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36988.log.gz 2024-09-14T23:27:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54382.log 2024-09-14T23:27:45.539 INFO:teuthology.orchestra.run.smithi089.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52389.log.gz 2024-09-14T23:27:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.72170.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72170.log.gz 2024-09-14T23:27:45.539 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66735.log.gz 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42279.log 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65344.log 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54382.log: /var/log/ceph/cluster2-client.mirror.42279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58277.log 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:gzip 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54382.log.gz 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65757.log 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58277.log: /var/log/ceph/cluster2-client.mirror.65344.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58277.log.gz 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45307.log 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65344.log.gz/var/log/ceph/cluster2-client.mirror.65757.log: 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68985.log 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45307.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45307.log.gz 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65757.log.gz 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52786.log 2024-09-14T23:27:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62413.log 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68985.log.gz/var/log/ceph/cluster1-client.mirror.52786.log: 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71473.log 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52786.log.gz 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43010.log 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62413.log: /var/log/ceph/cluster2-client.mirror.71473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61146.log 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43010.log: 87.3% 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71473.log.gz 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62413.log.gz 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43010.log.gz 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60096.log 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61146.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61146.log.gz 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70901.log 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61102.log 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70901.log: /var/log/ceph/cluster1-client.mirror.60096.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70901.log.gz 2024-09-14T23:27:45.541 INFO:teuthology.orchestra.run.smithi089.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60096.log.gz 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66111.log 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41674.log 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61102.log.gz 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.66111.log: --verbose -- /var/log/ceph/cluster2-client.mirror.47812.log 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45934.log 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41674.log: /var/log/ceph/cluster2-client.mirror.47812.log: 88.3% 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66111.log.gz 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41674.log.gz 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52606.log 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65613.log 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45934.log.gz 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52606.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49546.log 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65613.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65613.log.gz 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65302.log 2024-09-14T23:27:45.542 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49546.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49011.log.gz 2024-09-14T23:27:45.543 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52606.log.gz 2024-09-14T23:27:45.543 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49546.log.gz 2024-09-14T23:27:45.557 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70880.log 2024-09-14T23:27:45.557 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49380.log 2024-09-14T23:27:45.557 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65302.log.gz 2024-09-14T23:27:45.557 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58458.log 2024-09-14T23:27:45.557 INFO:teuthology.orchestra.run.smithi089.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70880.log.gz/var/log/ceph/cluster2-client.mirror.49380.log: 2024-09-14T23:27:45.557 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63294.log 2024-09-14T23:27:45.557 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58458.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49380.log.gz 2024-09-14T23:27:45.557 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65716.log 2024-09-14T23:27:45.557 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63294.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63294.log.gz 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45891.log 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58458.log.gz 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55268.log 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67192.log 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45891.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65716.log.gz/var/log/ceph/cluster1-client.mirror.55268.log: 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45891.log.gz 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55268.log.gz 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70549.log 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.67192.log: -- /var/log/ceph/cluster2-client.mirror.40998.log 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67192.log.gz 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70549.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70549.log.gz 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40745.log 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53723.log 2024-09-14T23:27:45.558 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48278.log 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64020.log 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53723.log.gz 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48278.log: gzip -5 --verbose -- 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48278.log.gz 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr: /var/log/ceph/cluster1-client.mirror.63390.log 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60120.log 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63390.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63390.log.gz 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64020.log.gz 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46850.log 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53429.log 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60120.log.gz 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47212.log 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46850.log.gz 2024-09-14T23:27:45.559 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41132.log 2024-09-14T23:27:45.560 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47212.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40998.log.gz 2024-09-14T23:27:45.560 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53429.log.gz 2024-09-14T23:27:45.560 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52449.log 2024-09-14T23:27:45.560 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49970.log 2024-09-14T23:27:45.560 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73660.log 2024-09-14T23:27:45.560 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49970.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41132.log.gz 2024-09-14T23:27:45.560 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52449.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52449.log.gz 2024-09-14T23:27:45.574 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72441.log 2024-09-14T23:27:45.575 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73660.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73660.log.gz 2024-09-14T23:27:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60373.log 2024-09-14T23:27:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72441.log: 87.7% 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40745.log.gz 2024-09-14T23:27:45.590 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.72441.log.gz 2024-09-14T23:27:45.594 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67417.log 2024-09-14T23:27:45.594 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60373.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60373.log.gz 2024-09-14T23:27:45.607 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64413.log 2024-09-14T23:27:45.615 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67417.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67417.log.gz 2024-09-14T23:27:45.616 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.62023.log 2024-09-14T23:27:45.616 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64413.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64413.log.gz 2024-09-14T23:27:45.617 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65654.log 2024-09-14T23:27:45.630 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.2.62023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67484.log 2024-09-14T23:27:45.631 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65654.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65654.log.gz 2024-09-14T23:27:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36899.log 2024-09-14T23:27:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67484.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67484.log.gz 2024-09-14T23:27:45.647 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45216.log 2024-09-14T23:27:45.651 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.36899.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42279.log.gz 2024-09-14T23:27:45.659 INFO:teuthology.orchestra.run.smithi089.stderr: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36899.log.gz 2024-09-14T23:27:45.661 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50740.log 2024-09-14T23:27:45.662 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45216.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45216.log.gz 2024-09-14T23:27:45.681 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40150.log 2024-09-14T23:27:45.681 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50740.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50740.log.gz 2024-09-14T23:27:45.691 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40066.log 2024-09-14T23:27:45.703 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40150.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.62023.log.gz 2024-09-14T23:27:45.705 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63414.log 2024-09-14T23:27:45.706 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45724.log 2024-09-14T23:27:45.706 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63414.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63414.log.gz 2024-09-14T23:27:45.711 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40066.log.gz 2024-09-14T23:27:45.711 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56366.log 2024-09-14T23:27:45.711 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45724.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45724.log.gz 2024-09-14T23:27:45.712 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67597.log 2024-09-14T23:27:45.712 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56366.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56366.log.gz 2024-09-14T23:27:45.714 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49506.log.gz 2024-09-14T23:27:45.714 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73938.log 2024-09-14T23:27:45.714 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67597.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67597.log.gz 2024-09-14T23:27:45.716 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51311.log 2024-09-14T23:27:45.716 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73938.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73938.log.gz 2024-09-14T23:27:45.716 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57260.log 2024-09-14T23:27:45.716 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51311.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41780.log 2024-09-14T23:27:45.717 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57260.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57260.log.gz 2024-09-14T23:27:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48100.log 2024-09-14T23:27:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41780.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51311.log.gz 2024-09-14T23:27:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68047.log 2024-09-14T23:27:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48100.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-14T23:27:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68047.log: 86.9% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48100.log.gz 2024-09-14T23:27:45.721 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68047.log.gz 2024-09-14T23:27:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67687.log 2024-09-14T23:27:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63149.log 2024-09-14T23:27:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72400.log 2024-09-14T23:27:45.721 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9%/var/log/ceph/cluster1-client.mirror.63149.log: -- replaced with /var/log/ceph/cluster1-client.mirror.67687.log.gz 2024-09-14T23:27:45.722 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63149.log.gz 2024-09-14T23:27:45.722 INFO:teuthology.orchestra.run.smithi089.stderr: -- /var/log/ceph/cluster2-client.mirror.50857.log 2024-09-14T23:27:45.723 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56230.log 2024-09-14T23:27:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50857.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50857.log.gz 2024-09-14T23:27:45.725 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72400.log.gz 2024-09-14T23:27:45.725 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41780.log.gz 2024-09-14T23:27:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.37015.log 2024-09-14T23:27:45.727 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73979.log 2024-09-14T23:27:45.727 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.2.37015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49156.log 2024-09-14T23:27:45.734 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73979.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73979.log.gz 2024-09-14T23:27:45.734 INFO:teuthology.orchestra.run.smithi089.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56230.log.gz 2024-09-14T23:27:45.734 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66321.log 2024-09-14T23:27:45.734 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49156.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49156.log.gz 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54137.log 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36339.log 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54137.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66321.log.gz 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54137.log.gz 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37417.log 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36339.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36339.log.gz 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46178.log 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.37417.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47009.log 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46178.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37417.log.gz 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67619.log 2024-09-14T23:27:45.735 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47009.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47009.log.gz 2024-09-14T23:27:45.736 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64868.log 2024-09-14T23:27:45.739 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67619.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67619.log.gz 2024-09-14T23:27:45.744 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73770.log 2024-09-14T23:27:45.744 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64868.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64868.log.gz 2024-09-14T23:27:45.745 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75029.log 2024-09-14T23:27:45.747 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73770.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73770.log.gz 2024-09-14T23:27:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73791.log 2024-09-14T23:27:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75029.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75029.log.gz 2024-09-14T23:27:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73238.log 2024-09-14T23:27:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73791.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73791.log.gz 2024-09-14T23:27:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70175.log 2024-09-14T23:27:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73238.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73238.log.gz 2024-09-14T23:27:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35791.log 2024-09-14T23:27:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70175.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70175.log.gz 2024-09-14T23:27:45.762 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46030.log 2024-09-14T23:27:45.762 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35791.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35791.log.gz 2024-09-14T23:27:45.762 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61194.log 2024-09-14T23:27:45.762 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.46030.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46030.log.gz 2024-09-14T23:27:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39619.log 2024-09-14T23:27:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61194.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61194.log.gz 2024-09-14T23:27:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44221.log 2024-09-14T23:27:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54609.log 2024-09-14T23:27:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52252.log 2024-09-14T23:27:45.789 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54609.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44221.log.gz 2024-09-14T23:27:45.789 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42214.log 2024-09-14T23:27:45.789 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7%/var/log/ceph/cluster2-client.mirror.52252.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48656.log.gz 2024-09-14T23:27:45.789 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52252.log.gz 2024-09-14T23:27:45.792 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43166.log 2024-09-14T23:27:45.792 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45594.log 2024-09-14T23:27:45.792 INFO:teuthology.orchestra.run.smithi089.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54609.log.gz 2024-09-14T23:27:45.793 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41084.log 2024-09-14T23:27:45.793 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45594.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45594.log.gz 2024-09-14T23:27:45.793 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51110.log 2024-09-14T23:27:45.794 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47700.log 2024-09-14T23:27:45.794 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41084.log.gz 2024-09-14T23:27:45.795 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51110.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51110.log.gz 2024-09-14T23:27:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74566.log 2024-09-14T23:27:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47700.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47700.log.gz 2024-09-14T23:27:45.798 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71846.log 2024-09-14T23:27:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74566.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74566.log.gz 2024-09-14T23:27:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49260.log 2024-09-14T23:27:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71846.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71846.log.gz 2024-09-14T23:27:45.801 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66176.log 2024-09-14T23:27:45.801 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43166.log.gz 2024-09-14T23:27:45.802 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51202.log 2024-09-14T23:27:45.805 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66176.log: 90.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50027.log 2024-09-14T23:27:45.805 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51202.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51202.log.gz 2024-09-14T23:27:45.805 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66176.log.gz 2024-09-14T23:27:45.809 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41108.log 2024-09-14T23:27:45.809 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50027.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50027.log.gz 2024-09-14T23:27:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55762.log 2024-09-14T23:27:45.813 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41108.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40194.log 2024-09-14T23:27:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55762.log: 87.4% 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41108.log.gz 2024-09-14T23:27:45.818 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55762.log.gz 2024-09-14T23:27:45.818 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40899.log 2024-09-14T23:27:45.826 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68452.log 2024-09-14T23:27:45.827 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52979.log 2024-09-14T23:27:45.827 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68452.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68452.log.gz 2024-09-14T23:27:45.831 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40899.log.gz 2024-09-14T23:27:45.831 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49672.log 2024-09-14T23:27:45.831 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52979.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52979.log.gz 2024-09-14T23:27:45.834 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71232.log.gz 2024-09-14T23:27:45.835 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73187.log 2024-09-14T23:27:45.835 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49672.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49672.log.gz 2024-09-14T23:27:45.837 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71596.log 2024-09-14T23:27:45.837 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73187.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73187.log.gz 2024-09-14T23:27:45.837 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.36997.log 2024-09-14T23:27:45.838 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71596.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71596.log.gz 2024-09-14T23:27:45.838 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.68786.log 2024-09-14T23:27:45.838 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.1.36997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40653.log 2024-09-14T23:27:45.841 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.3.68786.log: 70.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.36997.log.gz 2024-09-14T23:27:45.847 INFO:teuthology.orchestra.run.smithi089.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.37015.log.gz 2024-09-14T23:27:45.847 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59409.log 2024-09-14T23:27:45.847 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37383.log 2024-09-14T23:27:45.847 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59409.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59409.log.gz 2024-09-14T23:27:45.847 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46616.log 2024-09-14T23:27:45.847 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.37383.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.68786.log.gz 2024-09-14T23:27:45.847 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35667.log 2024-09-14T23:27:45.848 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46616.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40653.log.gz 2024-09-14T23:27:45.849 INFO:teuthology.orchestra.run.smithi089.stderr: 66.9% -- replaced with /var/log/ceph/cluster1-client.admin.37383.log.gz 2024-09-14T23:27:45.849 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65447.log 2024-09-14T23:27:45.849 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35667.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35667.log.gz 2024-09-14T23:27:45.851 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67462.log 2024-09-14T23:27:45.853 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65447.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65447.log.gz 2024-09-14T23:27:45.853 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37451.log 2024-09-14T23:27:45.862 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67462.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67462.log.gz 2024-09-14T23:27:45.862 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44737.log 2024-09-14T23:27:45.863 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46616.log.gz 2024-09-14T23:27:45.863 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49948.log 2024-09-14T23:27:45.870 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44737.log: /var/log/ceph/cluster1-client.mirror.37451.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44737.log.gz 2024-09-14T23:27:45.870 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose 57.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37451.log.gz 2024-09-14T23:27:45.871 INFO:teuthology.orchestra.run.smithi089.stderr: -- /var/log/ceph/cluster2-client.mirror.55961.log 2024-09-14T23:27:45.871 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44287.log 2024-09-14T23:27:45.871 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.55961.log: /var/log/ceph/cluster2-client.mirror.49948.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49948.log.gz 2024-09-14T23:27:45.873 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72022.log 2024-09-14T23:27:45.873 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44287.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55961.log.gz 2024-09-14T23:27:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-14T23:27:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.72022.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72022.log.gz 2024-09-14T23:27:45.879 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44287.log.gz 2024-09-14T23:27:45.880 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54793.log 2024-09-14T23:27:45.880 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71716.log 2024-09-14T23:27:45.880 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54793.log: /var/log/ceph/cluster2-osd.1.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56049.log 2024-09-14T23:27:45.880 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54793.log.gz 2024-09-14T23:27:45.880 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50443.log 2024-09-14T23:27:45.880 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71716.log: /var/log/ceph/cluster2-client.mirror.56049.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71716.log.gz 2024-09-14T23:27:45.881 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47913.log 2024-09-14T23:27:45.883 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50443.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50443.log.gz 2024-09-14T23:27:45.883 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55034.log 2024-09-14T23:27:45.883 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9%/var/log/ceph/cluster2-client.mirror.47913.log: -- replaced with /var/log/ceph/cluster2-client.mirror.56049.log.gz 2024-09-14T23:27:45.883 INFO:teuthology.orchestra.run.smithi089.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47913.log.gz 2024-09-14T23:27:45.884 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64971.log 2024-09-14T23:27:45.884 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55034.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55034.log.gz 2024-09-14T23:27:45.885 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70733.log 2024-09-14T23:27:45.887 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64971.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64971.log.gz 2024-09-14T23:27:45.887 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72359.log 2024-09-14T23:27:45.887 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.70733.log: --verbose -- /var/log/ceph/cluster1-client.mirror.58253.log 2024-09-14T23:27:45.889 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72359.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70733.log.gz 2024-09-14T23:27:45.889 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72359.log.gz 2024-09-14T23:27:45.889 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66797.log 2024-09-14T23:27:45.889 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55403.log 2024-09-14T23:27:45.889 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58253.log.gz 2024-09-14T23:27:45.890 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66797.log: /var/log/ceph/cluster2-client.mirror.55403.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55403.log.gz 2024-09-14T23:27:45.892 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66797.log.gz 2024-09-14T23:27:45.892 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72519.log 2024-09-14T23:27:45.892 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69165.log 2024-09-14T23:27:45.893 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72519.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72519.log.gz 2024-09-14T23:27:45.893 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56832.log 2024-09-14T23:27:45.895 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69165.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69165.log.gz 2024-09-14T23:27:45.895 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44049.log 2024-09-14T23:27:45.895 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56832.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56832.log.gz 2024-09-14T23:27:45.896 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61769.log 2024-09-14T23:27:45.897 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67844.log 2024-09-14T23:27:45.898 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61769.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61769.log.gz 2024-09-14T23:27:45.901 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52907.log 2024-09-14T23:27:45.902 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67844.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67844.log.gz 2024-09-14T23:27:45.903 INFO:teuthology.orchestra.run.smithi089.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44049.log.gz 2024-09-14T23:27:45.903 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59795.log 2024-09-14T23:27:45.904 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52907.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52907.log.gz 2024-09-14T23:27:45.904 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71173.log 2024-09-14T23:27:45.904 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74443.log 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59795.log.gz 93.7% 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71173.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47212.log.gz 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71173.log.gz 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69894.log 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61620.log 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74443.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71387.log 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69894.log: /var/log/ceph/cluster1-client.mirror.61620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61391.log 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.71387.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71387.log.gz 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61620.log.gz 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69894.log.gz 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42847.log 2024-09-14T23:27:45.910 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71152.log 2024-09-14T23:27:45.911 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61391.log.gz 2024-09-14T23:27:45.911 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.42847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43829.log 2024-09-14T23:27:45.911 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.75190.log 2024-09-14T23:27:45.911 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.42847.log.gz 2024-09-14T23:27:45.911 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43829.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.55123.log 2024-09-14T23:27:45.913 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.3.75190.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71152.log.gz 2024-09-14T23:27:45.914 INFO:teuthology.orchestra.run.smithi089.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.75190.log.gz 2024-09-14T23:27:45.914 INFO:teuthology.orchestra.run.smithi089.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43829.log.gz 2024-09-14T23:27:45.914 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52190.log 2024-09-14T23:27:45.914 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73410.log 2024-09-14T23:27:45.914 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52190.log: /var/log/ceph/cluster1-client.mirror.0.55123.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52190.log.gz 2024-09-14T23:27:45.914 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43849.log 2024-09-14T23:27:45.916 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46892.log 2024-09-14T23:27:45.916 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43849.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73410.log.gz 2024-09-14T23:27:45.916 INFO:teuthology.orchestra.run.smithi089.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43849.log.gz 2024-09-14T23:27:45.916 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72757.log 2024-09-14T23:27:45.916 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63101.log 2024-09-14T23:27:45.917 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72757.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72757.log.gz 2024-09-14T23:27:45.919 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52168.log 2024-09-14T23:27:45.919 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63101.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63101.log.gz 2024-09-14T23:27:45.920 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57542.log 2024-09-14T23:27:45.922 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52168.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52168.log.gz 2024-09-14T23:27:45.931 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.55123.log.gz 2024-09-14T23:27:45.931 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72461.log 2024-09-14T23:27:45.932 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57542.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57542.log.gz 2024-09-14T23:27:45.936 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36297.log 2024-09-14T23:27:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73812.log 2024-09-14T23:27:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36297.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36297.log.gz 2024-09-14T23:27:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72461.log.gz 2024-09-14T23:27:45.940 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.1.43614.log 2024-09-14T23:27:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66467.log 2024-09-14T23:27:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.1.43614.log: /var/log/ceph/cluster1-client.mirror.73812.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73812.log.gz 2024-09-14T23:27:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51710.log 2024-09-14T23:27:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66467.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66467.log.gz 2024-09-14T23:27:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45913.log 2024-09-14T23:27:45.944 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51710.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51710.log.gz 2024-09-14T23:27:45.944 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43503.log 2024-09-14T23:27:45.959 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50209.log 2024-09-14T23:27:45.960 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43503.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43503.log.gz 2024-09-14T23:27:45.973 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66238.log 2024-09-14T23:27:45.984 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50209.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49970.log.gz 2024-09-14T23:27:45.989 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65551.log 2024-09-14T23:27:45.990 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66238.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66238.log.gz 2024-09-14T23:27:45.991 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60510.log 2024-09-14T23:27:45.992 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65551.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65551.log.gz 2024-09-14T23:27:46.007 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66968.log 2024-09-14T23:27:46.008 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60510.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60510.log.gz 2024-09-14T23:27:46.009 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42214.log.gz 2024-09-14T23:27:46.021 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59337.log 2024-09-14T23:27:46.021 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.66968.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66968.log.gz 2024-09-14T23:27:46.033 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59662.log 2024-09-14T23:27:46.034 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59337.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59337.log.gz 2024-09-14T23:27:46.047 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60460.log 2024-09-14T23:27:46.047 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59662.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59662.log.gz 2024-09-14T23:27:46.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53381.log 2024-09-14T23:27:46.060 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60460.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60460.log.gz 2024-09-14T23:27:46.066 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63704.log 2024-09-14T23:27:46.075 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53381.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62860.log 2024-09-14T23:27:46.079 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63704.log: -- replaced with /var/log/ceph/cluster1-client.mirror.53381.log.gz 2024-09-14T23:27:46.088 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54046.log 2024-09-14T23:27:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62860.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62860.log.gz 2024-09-14T23:27:46.089 INFO:teuthology.orchestra.run.smithi089.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63704.log.gz 2024-09-14T23:27:46.100 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36840.log 2024-09-14T23:27:46.103 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54046.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54046.log.gz 2024-09-14T23:27:46.109 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36507.log 2024-09-14T23:27:46.119 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.0.36840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40018.log 2024-09-14T23:27:46.120 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.36507.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36507.log.gz 2024-09-14T23:27:46.128 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48417.log 2024-09-14T23:27:46.131 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40018.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46499.log.gz 2024-09-14T23:27:46.132 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43963.log 2024-09-14T23:27:46.134 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.48417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44264.log 2024-09-14T23:27:46.138 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43963.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49260.log.gz 2024-09-14T23:27:46.139 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54858.log 2024-09-14T23:27:46.140 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44264.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43963.log.gz 2024-09-14T23:27:46.143 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40018.log.gz 2024-09-14T23:27:46.144 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75320.log 2024-09-14T23:27:46.144 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54858.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44264.log.gz 2024-09-14T23:27:46.145 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60325.log 2024-09-14T23:27:46.145 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75320.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75320.log.gz 2024-09-14T23:27:46.147 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53583.log 2024-09-14T23:27:46.147 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60325.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54858.log.gz 2024-09-14T23:27:46.147 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60325.log.gz 2024-09-14T23:27:46.147 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61122.log 2024-09-14T23:27:46.148 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53583.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53583.log.gz 2024-09-14T23:27:46.148 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59915.log 2024-09-14T23:27:46.148 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46236.log 2024-09-14T23:27:46.150 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59915.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61122.log.gz 2024-09-14T23:27:46.151 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59915.log.gz 2024-09-14T23:27:46.151 INFO:teuthology.orchestra.run.smithi089.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.43614.log.gz 2024-09-14T23:27:46.151 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57280.log 2024-09-14T23:27:46.151 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39218.log 2024-09-14T23:27:46.151 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46236.log.gz/var/log/ceph/cluster1-client.mirror.57280.log: 2024-09-14T23:27:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67214.log 2024-09-14T23:27:46.154 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57280.log.gz 2024-09-14T23:27:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.39218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36591.log 2024-09-14T23:27:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.67214.log: --verbose -- /var/log/ceph/cluster2-client.mirror.46258.log 2024-09-14T23:27:46.154 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9%/var/log/ceph/cluster2-client.admin.36591.log: -- replaced with /var/log/ceph/cluster1-client.mirror.67214.log.gz 2024-09-14T23:27:46.154 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36591.log.gz 2024-09-14T23:27:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74167.log 2024-09-14T23:27:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46258.log: /var/log/ceph/cluster2-client.mirror.74167.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74167.log.gz 2024-09-14T23:27:46.155 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39218.log.gz 2024-09-14T23:27:46.156 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65385.log 2024-09-14T23:27:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69278.log 2024-09-14T23:27:46.158 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65385.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65385.log.gz 2024-09-14T23:27:46.160 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64992.log 2024-09-14T23:27:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.69278.log: /var/log/ceph/cluster2-client.mirror.39435.log 2024-09-14T23:27:46.162 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69278.log.gz 2024-09-14T23:27:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44760.log 2024-09-14T23:27:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39435.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64992.log.gz 2024-09-14T23:27:46.163 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39435.log.gz 2024-09-14T23:27:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43873.log 2024-09-14T23:27:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45492.log 2024-09-14T23:27:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43873.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43873.log.gz 2024-09-14T23:27:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39661.log 2024-09-14T23:27:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45492.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45492.log.gz 2024-09-14T23:27:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44373.log 2024-09-14T23:27:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46971.log 2024-09-14T23:27:46.171 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44373.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44760.log.gz 2024-09-14T23:27:46.171 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58848.log 2024-09-14T23:27:46.175 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46971.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44373.log.gz 2024-09-14T23:27:46.179 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39661.log.gz 2024-09-14T23:27:46.184 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57590.log 2024-09-14T23:27:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58848.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58848.log.gz 2024-09-14T23:27:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73307.log 2024-09-14T23:27:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57590.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57590.log.gz 2024-09-14T23:27:46.189 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46418.log 2024-09-14T23:27:46.189 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73307.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73307.log.gz 2024-09-14T23:27:46.190 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44883.log 2024-09-14T23:27:46.193 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46418.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48417.log.gz 2024-09-14T23:27:46.210 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60165.log 2024-09-14T23:27:46.210 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47812.log.gz 2024-09-14T23:27:46.211 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44883.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44883.log.gz 2024-09-14T23:27:46.212 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52469.log 2024-09-14T23:27:46.214 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60165.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60165.log.gz 2024-09-14T23:27:46.214 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45184.log 2024-09-14T23:27:46.218 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52469.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52469.log.gz 2024-09-14T23:27:46.221 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71452.log 2024-09-14T23:27:46.222 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45184.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45184.log.gz 2024-09-14T23:27:46.223 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36717.log 2024-09-14T23:27:46.224 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.71452.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71452.log.gz 2024-09-14T23:27:46.224 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43082.log 2024-09-14T23:27:46.224 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36717.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36717.log.gz 2024-09-14T23:27:46.225 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.59092.log 2024-09-14T23:27:46.226 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43082.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43082.log.gz 2024-09-14T23:27:46.226 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50812.log 2024-09-14T23:27:46.246 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.0.59092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48842.log 2024-09-14T23:27:46.247 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50812.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50812.log.gz 2024-09-14T23:27:46.253 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41832.log 2024-09-14T23:27:46.262 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69347.log 2024-09-14T23:27:46.267 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.41832.log: /var/log/ceph/cluster2-client.mirror.48842.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.59092.log.gz 2024-09-14T23:27:46.275 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.41832.log.gz 2024-09-14T23:27:46.278 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61885.log 2024-09-14T23:27:46.279 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69347.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69347.log.gz 2024-09-14T23:27:46.284 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46178.log.gz 2024-09-14T23:27:46.285 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54697.log 2024-09-14T23:27:46.287 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61885.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61885.log.gz 2024-09-14T23:27:46.287 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47089.log 2024-09-14T23:27:46.287 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54697.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54697.log.gz 2024-09-14T23:27:46.288 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74058.log 2024-09-14T23:27:46.288 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47089.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47089.log.gz 2024-09-14T23:27:46.288 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40793.log 2024-09-14T23:27:46.290 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74058.log: 82.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74058.log.gz 2024-09-14T23:27:46.290 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42900.log 2024-09-14T23:27:46.291 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63591.log 2024-09-14T23:27:46.292 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42900.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42900.log.gz 2024-09-14T23:27:46.293 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40793.log.gz 2024-09-14T23:27:46.293 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64764.log 2024-09-14T23:27:46.294 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63591.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63591.log.gz 2024-09-14T23:27:46.294 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47393.log 2024-09-14T23:27:46.296 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64764.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64764.log.gz 2024-09-14T23:27:46.298 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64000.log 2024-09-14T23:27:46.298 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47393.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47393.log.gz 2024-09-14T23:27:46.298 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68820.log 2024-09-14T23:27:46.299 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64000.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64000.log.gz 2024-09-14T23:27:46.299 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49447.log 2024-09-14T23:27:46.300 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68820.log: 86.9% 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68820.log.gz 2024-09-14T23:27:46.300 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41221.log.gz 2024-09-14T23:27:46.300 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53478.log 2024-09-14T23:27:46.301 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49447.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49447.log.gz 2024-09-14T23:27:46.301 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72798.log 2024-09-14T23:27:46.302 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53478.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53478.log.gz 2024-09-14T23:27:46.302 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46929.log 2024-09-14T23:27:46.303 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72798.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72798.log.gz 2024-09-14T23:27:46.303 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67372.log 2024-09-14T23:27:46.304 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46929.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46929.log.gz 2024-09-14T23:27:46.305 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39946.log 2024-09-14T23:27:46.305 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67372.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67372.log.gz 2024-09-14T23:27:46.306 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63771.log 2024-09-14T23:27:46.306 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52324.log 2024-09-14T23:27:46.308 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63771.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63771.log.gz 2024-09-14T23:27:46.308 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57037.log 2024-09-14T23:27:46.309 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39946.log.gz 2024-09-14T23:27:46.309 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52324.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52324.log.gz 2024-09-14T23:27:46.310 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71883.log 2024-09-14T23:27:46.310 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57037.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57037.log.gz 2024-09-14T23:27:46.311 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46520.log 2024-09-14T23:27:46.311 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71883.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71883.log.gz 2024-09-14T23:27:46.312 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65468.log 2024-09-14T23:27:46.312 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46520.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46520.log.gz 2024-09-14T23:27:46.313 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56543.log 2024-09-14T23:27:46.313 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65468.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65468.log.gz 2024-09-14T23:27:46.314 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55714.log 2024-09-14T23:27:46.314 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.56543.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56543.log.gz 2024-09-14T23:27:46.315 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67642.log 2024-09-14T23:27:46.316 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.55714.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55714.log.gz 2024-09-14T23:27:46.316 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60713.log 2024-09-14T23:27:46.317 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67642.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67642.log.gz 2024-09-14T23:27:46.317 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71678.log 2024-09-14T23:27:46.318 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60713.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60713.log.gz 2024-09-14T23:27:46.318 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50559.log 2024-09-14T23:27:46.319 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71678.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71678.log.gz 2024-09-14T23:27:46.319 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57898.log 2024-09-14T23:27:46.320 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50559.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50559.log.gz 2024-09-14T23:27:46.320 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73557.log 2024-09-14T23:27:46.321 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57898.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57898.log.gz 2024-09-14T23:27:46.322 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53245.log 2024-09-14T23:27:46.323 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73557.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73557.log.gz 2024-09-14T23:27:46.324 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67754.log 2024-09-14T23:27:46.325 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53245.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53245.log.gz 2024-09-14T23:27:46.326 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66632.log 2024-09-14T23:27:46.327 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67754.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67754.log.gz 2024-09-14T23:27:46.328 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61552.log 2024-09-14T23:27:46.328 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66632.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66632.log.gz 2024-09-14T23:27:46.329 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36633.log 2024-09-14T23:27:46.330 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61552.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61552.log.gz 2024-09-14T23:27:46.331 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69188.log 2024-09-14T23:27:46.331 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36633.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36633.log.gz 2024-09-14T23:27:46.333 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71930.log 2024-09-14T23:27:46.333 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69188.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69188.log.gz 2024-09-14T23:27:46.333 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73259.log 2024-09-14T23:27:46.334 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.71930.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71930.log.gz 2024-09-14T23:27:46.334 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52827.log 2024-09-14T23:27:46.335 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73259.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73259.log.gz 2024-09-14T23:27:46.335 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52586.log 2024-09-14T23:27:46.338 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52827.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52827.log.gz 2024-09-14T23:27:46.338 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54247.log 2024-09-14T23:27:46.340 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52586.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52586.log.gz 2024-09-14T23:27:46.341 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52762.log 2024-09-14T23:27:46.342 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.54247.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54247.log.gz 2024-09-14T23:27:46.342 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54880.log 2024-09-14T23:27:46.343 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52762.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52762.log.gz 2024-09-14T23:27:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43188.log 2024-09-14T23:27:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54880.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54880.log.gz 2024-09-14T23:27:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32443.log 2024-09-14T23:27:46.346 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.43188.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43188.log.gz 2024-09-14T23:27:46.347 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52670.log 2024-09-14T23:27:46.347 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.32443.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32443.log.gz 2024-09-14T23:27:46.348 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41333.log 2024-09-14T23:27:46.349 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52670.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52670.log.gz 2024-09-14T23:27:46.349 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72482.log 2024-09-14T23:27:46.350 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.41333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48888.log 2024-09-14T23:27:46.351 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72482.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72482.log.gz 2024-09-14T23:27:46.351 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46892.log.gz 2024-09-14T23:27:46.352 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62527.log 2024-09-14T23:27:46.353 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48888.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48888.log.gz 2024-09-14T23:27:46.354 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41333.log.gz 2024-09-14T23:27:46.356 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61861.log 2024-09-14T23:27:46.356 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72631.log 2024-09-14T23:27:46.357 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61861.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61861.log.gz 2024-09-14T23:27:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57085.log 2024-09-14T23:27:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.72631.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72631.log.gz 2024-09-14T23:27:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46804.log 2024-09-14T23:27:46.361 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57085.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57085.log.gz 2024-09-14T23:27:46.361 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40150.log.gz 2024-09-14T23:27:46.361 INFO:teuthology.orchestra.run.smithi089.stderr: /var/log/ceph/cluster2-client.mirror.58147.log 2024-09-14T23:27:46.362 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46804.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62527.log.gz 2024-09-14T23:27:46.362 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38921.log 2024-09-14T23:27:46.364 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58147.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58147.log.gz 2024-09-14T23:27:46.364 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50463.log 2024-09-14T23:27:46.364 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.38921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42257.log 2024-09-14T23:27:46.365 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39619.log.gz 2024-09-14T23:27:46.366 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50463.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50463.log.gz 2024-09-14T23:27:46.368 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67079.log 2024-09-14T23:27:46.368 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69730.log 2024-09-14T23:27:46.368 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67079.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67079.log.gz 2024-09-14T23:27:46.368 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45509.log 2024-09-14T23:27:46.369 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69730.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69730.log.gz 2024-09-14T23:27:46.369 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46477.log 2024-09-14T23:27:46.371 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45509.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45509.log.gz 2024-09-14T23:27:46.371 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73390.log 2024-09-14T23:27:46.371 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46477.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46477.log.gz 2024-09-14T23:27:46.373 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37349.log 2024-09-14T23:27:46.373 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73390.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73390.log.gz 2024-09-14T23:27:46.373 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49299.log 2024-09-14T23:27:46.381 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.37349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50604.log 2024-09-14T23:27:46.382 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49299.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49299.log.gz 2024-09-14T23:27:46.384 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59313.log 2024-09-14T23:27:46.387 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50604.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50604.log.gz 2024-09-14T23:27:46.387 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42033.log 2024-09-14T23:27:46.387 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59313.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59313.log.gz 2024-09-14T23:27:46.389 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63214.log 2024-09-14T23:27:46.389 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49739.log 2024-09-14T23:27:46.391 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63214.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63214.log.gz 2024-09-14T23:27:46.392 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57811.log 2024-09-14T23:27:46.392 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.49739.log: 94.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49739.log.gz 2024-09-14T23:27:46.393 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60929.log 2024-09-14T23:27:46.394 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57811.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57811.log.gz 2024-09-14T23:27:46.395 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55561.log 2024-09-14T23:27:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60929.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60929.log.gz 2024-09-14T23:27:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56414.log 2024-09-14T23:27:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55561.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55561.log.gz 2024-09-14T23:27:46.397 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35790.log 2024-09-14T23:27:46.402 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56414.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56414.log.gz 2024-09-14T23:27:46.402 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58687.log 2024-09-14T23:27:46.402 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35790.log.gz 2024-09-14T23:27:46.403 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75514.log 2024-09-14T23:27:46.403 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58687.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58687.log.gz 2024-09-14T23:27:46.404 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.40485.log 2024-09-14T23:27:46.405 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.75514.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75514.log.gz 2024-09-14T23:27:46.406 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48524.log 2024-09-14T23:27:46.406 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.0.40485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63938.log 2024-09-14T23:27:46.406 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38921.log.gz 2024-09-14T23:27:46.408 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48524.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48524.log.gz 2024-09-14T23:27:46.408 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37646.log 2024-09-14T23:27:46.408 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63938.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63938.log.gz 2024-09-14T23:27:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75340.log 2024-09-14T23:27:46.414 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.37646.log: 67.2% -- replaced with /var/log/ceph/cluster2-client.admin.37646.log.gz 2024-09-14T23:27:46.414 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50353.log 2024-09-14T23:27:46.416 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.75340.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75340.log.gz 2024-09-14T23:27:46.416 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40857.log 2024-09-14T23:27:46.417 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50353.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50353.log.gz 2024-09-14T23:27:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53699.log 2024-09-14T23:27:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44479.log 2024-09-14T23:27:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53699.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53699.log.gz 2024-09-14T23:27:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67732.log 2024-09-14T23:27:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44479.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50209.log.gz 2024-09-14T23:27:46.429 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44479.log.gz 2024-09-14T23:27:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74288.log 2024-09-14T23:27:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67732.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42257.log.gz 2024-09-14T23:27:46.433 INFO:teuthology.orchestra.run.smithi089.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67732.log.gz 2024-09-14T23:27:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37502.log 2024-09-14T23:27:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74879.log 2024-09-14T23:27:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.37502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64826.log 2024-09-14T23:27:46.433 INFO:teuthology.orchestra.run.smithi089.stderr: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37502.log.gz 2024-09-14T23:27:46.434 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.74879.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74879.log.gz 2024-09-14T23:27:46.435 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71429.log 2024-09-14T23:27:46.436 INFO:teuthology.orchestra.run.smithi089.stderr: 79.7% -- replaced with /var/log/ceph/cluster1-client.admin.37349.log.gz 2024-09-14T23:27:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64826.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64826.log.gz 2024-09-14T23:27:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50969.log 2024-09-14T23:27:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.71429.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71429.log.gz 2024-09-14T23:27:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62736.log 2024-09-14T23:27:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50969.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50969.log.gz 2024-09-14T23:27:46.440 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42372.log 2024-09-14T23:27:46.441 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62736.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62736.log.gz 2024-09-14T23:27:46.442 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55010.log 2024-09-14T23:27:46.443 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.42372.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42372.log.gz 2024-09-14T23:27:46.443 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68227.log 2024-09-14T23:27:46.445 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55010.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55010.log.gz 2024-09-14T23:27:46.445 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45746.log 2024-09-14T23:27:46.445 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68227.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68227.log.gz 2024-09-14T23:27:46.447 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63543.log 2024-09-14T23:27:46.464 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45746.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40857.log.gz 2024-09-14T23:27:46.464 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45653.log 2024-09-14T23:27:46.464 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63543.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63543.log.gz 2024-09-14T23:27:46.475 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53152.log 2024-09-14T23:27:46.494 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37366.log 2024-09-14T23:27:46.508 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53152.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53152.log.gz 2024-09-14T23:27:46.513 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56165.log 2024-09-14T23:27:46.518 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.37366.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.37366.log.gz 2024-09-14T23:27:46.526 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53405.log 2024-09-14T23:27:46.527 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56165.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56165.log.gz 2024-09-14T23:27:46.537 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47834.log 2024-09-14T23:27:46.537 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53405.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53405.log.gz 2024-09-14T23:27:46.545 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71617.log 2024-09-14T23:27:46.554 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45687.log 2024-09-14T23:27:46.554 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71617.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71617.log.gz 2024-09-14T23:27:46.557 INFO:teuthology.orchestra.run.smithi089.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47834.log.gz 2024-09-14T23:27:46.559 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48842.log.gz 2024-09-14T23:27:46.559 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42033.log.gz 2024-09-14T23:27:46.559 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47452.log 2024-09-14T23:27:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45687.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45687.log.gz 2024-09-14T23:27:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60792.log 2024-09-14T23:27:46.571 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76040.log 2024-09-14T23:27:46.571 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60792.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45913.log.gz 2024-09-14T23:27:46.573 INFO:teuthology.orchestra.run.smithi089.stderr: 94.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60792.log.gz 2024-09-14T23:27:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68497.log 2024-09-14T23:27:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.76040.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76040.log.gz 2024-09-14T23:27:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52276.log 2024-09-14T23:27:46.576 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68497.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68497.log.gz 2024-09-14T23:27:46.576 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55445.log 2024-09-14T23:27:46.576 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52276.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52276.log.gz 2024-09-14T23:27:46.576 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48083.log 2024-09-14T23:27:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.55445.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55445.log.gz 2024-09-14T23:27:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55340.log 2024-09-14T23:27:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48083.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48083.log.gz 2024-09-14T23:27:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71535.log 2024-09-14T23:27:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55340.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55340.log.gz 2024-09-14T23:27:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61042.log 2024-09-14T23:27:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71535.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71535.log.gz 2024-09-14T23:27:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58105.log 2024-09-14T23:27:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61042.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61042.log.gz 2024-09-14T23:27:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69382.log 2024-09-14T23:27:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58105.log: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.40485.log.gz 2024-09-14T23:27:46.586 INFO:teuthology.orchestra.run.smithi089.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58105.log.gz 2024-09-14T23:27:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62271.log 2024-09-14T23:27:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69382.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69382.log.gz 2024-09-14T23:27:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40336.log 2024-09-14T23:27:46.589 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62271.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62271.log.gz 2024-09-14T23:27:46.589 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55361.log 2024-09-14T23:27:46.590 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.40336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70237.log 2024-09-14T23:27:46.593 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.55361.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55361.log.gz 2024-09-14T23:27:46.593 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68114.log 2024-09-14T23:27:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70237.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70237.log.gz 2024-09-14T23:27:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47430.log 2024-09-14T23:27:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68114.log: 86.9% 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.40336.log.gz 2024-09-14T23:27:46.596 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68114.log.gz 2024-09-14T23:27:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72499.log 2024-09-14T23:27:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47430.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47430.log.gz 2024-09-14T23:27:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63173.log 2024-09-14T23:27:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.72499.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72499.log.gz 2024-09-14T23:27:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60144.log 2024-09-14T23:27:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63173.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63173.log.gz 2024-09-14T23:27:46.600 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59590.log 2024-09-14T23:27:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60144.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60144.log.gz 2024-09-14T23:27:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65012.log 2024-09-14T23:27:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59590.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59590.log.gz 2024-09-14T23:27:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64206.log 2024-09-14T23:27:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65012.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65012.log.gz 2024-09-14T23:27:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57470.log 2024-09-14T23:27:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64206.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64206.log.gz 2024-09-14T23:27:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63455.log 2024-09-14T23:27:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57470.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57470.log.gz 2024-09-14T23:27:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50229.log 2024-09-14T23:27:46.612 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63455.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63455.log.gz 2024-09-14T23:27:46.612 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68654.log 2024-09-14T23:27:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50229.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50229.log.gz 2024-09-14T23:27:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67664.log 2024-09-14T23:27:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68654.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68654.log.gz 2024-09-14T23:27:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53066.log 2024-09-14T23:27:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67664.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67664.log.gz 2024-09-14T23:27:46.616 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36951.log 2024-09-14T23:27:46.616 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53066.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53066.log.gz 2024-09-14T23:27:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49238.log 2024-09-14T23:27:46.618 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.36951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61986.log 2024-09-14T23:27:46.618 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49238.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49238.log.gz 2024-09-14T23:27:46.619 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60977.log 2024-09-14T23:27:46.621 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61986.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61986.log.gz 2024-09-14T23:27:46.621 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 81.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36951.log.gz 2024-09-14T23:27:46.621 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.40673.log 2024-09-14T23:27:46.622 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.60977.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60977.log.gz 2024-09-14T23:27:46.622 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61926.log 2024-09-14T23:27:46.623 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.40673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72654.log 2024-09-14T23:27:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61926.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61926.log.gz 2024-09-14T23:27:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64744.log 2024-09-14T23:27:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.72654.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72654.log.gz 2024-09-14T23:27:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55316.log 2024-09-14T23:27:46.627 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40673.log.gz 2024-09-14T23:27:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64744.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64744.log.gz 2024-09-14T23:27:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75746.log 2024-09-14T23:27:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55316.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55316.log.gz 2024-09-14T23:27:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53499.log 2024-09-14T23:27:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.75746.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75746.log.gz 2024-09-14T23:27:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61323.log 2024-09-14T23:27:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53499.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53499.log.gz 2024-09-14T23:27:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49464.log 2024-09-14T23:27:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61323.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61323.log.gz 2024-09-14T23:27:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75998.log 2024-09-14T23:27:46.637 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49464.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49464.log.gz 2024-09-14T23:27:46.637 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69792.log 2024-09-14T23:27:46.638 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.75998.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75998.log.gz 2024-09-14T23:27:46.638 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45003.log 2024-09-14T23:27:46.640 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69792.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69792.log.gz 2024-09-14T23:27:46.640 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62462.log 2024-09-14T23:27:46.640 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45003.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45003.log.gz 2024-09-14T23:27:46.641 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51569.log 2024-09-14T23:27:46.642 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62462.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62462.log.gz 2024-09-14T23:27:46.643 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49990.log 2024-09-14T23:27:46.644 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51569.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51569.log.gz 2024-09-14T23:27:46.644 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70586.log 2024-09-14T23:27:46.645 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49990.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49990.log.gz 2024-09-14T23:27:46.645 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71190.log 2024-09-14T23:27:46.646 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70586.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70586.log.gz 2024-09-14T23:27:46.646 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53357.log 2024-09-14T23:27:46.647 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71190.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71190.log.gz 2024-09-14T23:27:46.648 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64723.log 2024-09-14T23:27:46.649 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53357.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53357.log.gz 2024-09-14T23:27:46.649 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48930.log 2024-09-14T23:27:46.650 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64723.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64723.log.gz 2024-09-14T23:27:46.651 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73349.log 2024-09-14T23:27:46.661 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47680.log 2024-09-14T23:27:46.662 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73349.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73349.log.gz 2024-09-14T23:27:46.668 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73085.log 2024-09-14T23:27:46.679 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47680.log: 88.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57126.log 2024-09-14T23:27:46.679 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47680.log.gz 2024-09-14T23:27:46.679 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73085.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73085.log.gz 2024-09-14T23:27:46.686 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46804.log.gz 2024-09-14T23:27:46.691 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46971.log.gz 2024-09-14T23:27:46.695 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50187.log 2024-09-14T23:27:46.718 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57126.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57126.log.gz 2024-09-14T23:27:46.718 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71863.log 2024-09-14T23:27:46.726 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50187.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50187.log.gz 2024-09-14T23:27:46.727 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66694.log 2024-09-14T23:27:46.735 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71863.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71863.log.gz 2024-09-14T23:27:46.735 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74902.log 2024-09-14T23:27:46.736 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66694.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66694.log.gz 2024-09-14T23:27:46.736 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66070.log 2024-09-14T23:27:46.736 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.74902.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74902.log.gz 2024-09-14T23:27:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73877.log 2024-09-14T23:27:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66070.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66070.log.gz 2024-09-14T23:27:46.749 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66132.log 2024-09-14T23:27:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73877.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73877.log.gz 2024-09-14T23:27:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55244.log 2024-09-14T23:27:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66132.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66132.log.gz 2024-09-14T23:27:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67799.log 2024-09-14T23:27:46.763 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55244.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55244.log.gz 2024-09-14T23:27:46.766 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47170.log 2024-09-14T23:27:46.770 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67799.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67799.log.gz 2024-09-14T23:27:46.770 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73578.log 2024-09-14T23:27:46.771 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47170.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47170.log.gz 2024-09-14T23:27:46.772 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62756.log 2024-09-14T23:27:46.781 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73578.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73578.log.gz 2024-09-14T23:27:46.781 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52493.log 2024-09-14T23:27:46.782 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62756.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62756.log.gz 2024-09-14T23:27:46.783 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63639.log 2024-09-14T23:27:46.784 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52493.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52493.log.gz 2024-09-14T23:27:46.784 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.74785.log 2024-09-14T23:27:46.785 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.63639.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63639.log.gz 2024-09-14T23:27:46.785 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68722.log 2024-09-14T23:27:46.795 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.0.74785.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.74785.log.gz 2024-09-14T23:27:46.795 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42194.log 2024-09-14T23:27:46.798 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68722.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68722.log.gz 2024-09-14T23:27:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69627.log 2024-09-14T23:27:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52738.log 2024-09-14T23:27:46.801 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69627.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69627.log.gz 2024-09-14T23:27:46.801 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73835.log 2024-09-14T23:27:46.801 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52738.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52738.log.gz 2024-09-14T23:27:46.801 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75368.log 2024-09-14T23:27:46.806 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.73835.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46258.log.gz 2024-09-14T23:27:46.806 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73835.log.gz 2024-09-14T23:27:46.806 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59430.log 2024-09-14T23:27:46.809 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.75368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52517.log 2024-09-14T23:27:46.809 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75368.log.gz 2024-09-14T23:27:46.810 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55648.log 2024-09-14T23:27:46.810 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52517.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52517.log.gz 2024-09-14T23:27:46.810 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71555.log 2024-09-14T23:27:46.810 INFO:teuthology.orchestra.run.smithi089.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59430.log.gz 2024-09-14T23:27:46.810 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.55648.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-14T23:27:46.810 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55648.log.gz 2024-09-14T23:27:46.812 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71110.log 2024-09-14T23:27:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2.log: 93.5% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45746.log.gz 2024-09-14T23:27:46.816 INFO:teuthology.orchestra.run.smithi089.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36840.log.gz 2024-09-14T23:27:46.816 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42194.log.gz 2024-09-14T23:27:46.816 INFO:teuthology.orchestra.run.smithi089.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71555.log.gz 2024-09-14T23:27:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59891.log 2024-09-14T23:27:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71110.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44201.log 2024-09-14T23:27:46.816 INFO:teuthology.orchestra.run.smithi089.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71110.log.gz 2024-09-14T23:27:46.817 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58125.log 2024-09-14T23:27:46.817 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.44201.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59891.log.gz 2024-09-14T23:27:46.817 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46316.log 2024-09-14T23:27:46.819 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58125.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58125.log.gz 2024-09-14T23:27:46.819 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65219.log 2024-09-14T23:27:46.823 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46316.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46316.log.gz 2024-09-14T23:27:46.823 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.74058.log 2024-09-14T23:27:46.823 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65219.log: 87.2% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-14T23:27:46.823 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44201.log.gz 2024-09-14T23:27:46.823 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65219.log.gz 2024-09-14T23:27:46.823 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66756.log 2024-09-14T23:27:46.823 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.1.74058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61731.log 2024-09-14T23:27:46.824 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62184.log 2024-09-14T23:27:46.824 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66756.log.gz 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61731.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61731.log.gz 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58599.log 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62184.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62184.log.gz 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.41465.log 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.58599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63792.log 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.74058.log.gz 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.3.41465.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66818.log 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45282.log 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63792.log.gz 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49119.log 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.45282.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66818.log.gz 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45282.log.gz 2024-09-14T23:27:46.829 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:46.830 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71905.log 2024-09-14T23:27:46.830 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49119.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49119.log.gz 2024-09-14T23:27:46.830 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58599.log.gz 2024-09-14T23:27:46.833 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36149.log 2024-09-14T23:27:46.833 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.71905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51152.log 2024-09-14T23:27:46.833 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36149.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36149.log.gz 2024-09-14T23:27:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38965.log 2024-09-14T23:27:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51152.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51152.log.gz 2024-09-14T23:27:46.848 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74288.log.gz 2024-09-14T23:27:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69539.log 2024-09-14T23:27:46.850 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.38965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67237.log 2024-09-14T23:27:46.850 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46279.log 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69539.log: /var/log/ceph/cluster1-client.mirror.67237.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69539.log.gz 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67237.log.gz 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54179.log 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65778.log 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46279.log.gz 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63615.log 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40945.log 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65778.log: /var/log/ceph/cluster1-client.mirror.63615.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54179.log.gz 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65778.log.gz 2024-09-14T23:27:46.853 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63615.log.gz 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.40945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43261.log 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40697.log 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.43261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51794.log 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.43261.log.gz 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41201.log 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.51794.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38965.log.gz 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.40945.log.gz 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51794.log.gz 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41578.log 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.41201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48031.log 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41578.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46579.log 2024-09-14T23:27:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48031.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40697.log.gz 2024-09-14T23:27:46.860 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48031.log.gz 2024-09-14T23:27:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50949.log 2024-09-14T23:27:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67147.log 2024-09-14T23:27:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62229.log 2024-09-14T23:27:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50949.log: /var/log/ceph/cluster1-client.mirror.67147.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67147.log.gz 2024-09-14T23:27:46.862 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50949.log.gz 2024-09-14T23:27:46.869 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62884.log 2024-09-14T23:27:46.872 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.62229.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62229.log.gz 2024-09-14T23:27:46.882 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41715.log 2024-09-14T23:27:46.886 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.62884.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62884.log.gz 2024-09-14T23:27:46.886 INFO:teuthology.orchestra.run.smithi089.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.3.41465.log.gz 2024-09-14T23:27:46.886 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41264.log 2024-09-14T23:27:46.888 INFO:teuthology.orchestra.run.smithi089.stderr:gzip/var/log/ceph/cluster1-client.mirror.41715.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61906.log 2024-09-14T23:27:46.888 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42648.log 2024-09-14T23:27:46.888 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71364.log 2024-09-14T23:27:46.894 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42648.log: /var/log/ceph/cluster2-client.mirror.61906.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42648.log.gz 2024-09-14T23:27:46.894 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53786.log 2024-09-14T23:27:46.898 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.71364.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61906.log.gz 2024-09-14T23:27:46.898 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71364.log.gz 2024-09-14T23:27:46.898 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36856.log 2024-09-14T23:27:46.898 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.53786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36023.log 2024-09-14T23:27:46.898 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53786.log.gz 2024-09-14T23:27:46.899 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.36856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48783.log 2024-09-14T23:27:46.899 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66029.log 2024-09-14T23:27:46.899 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48783.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48783.log.gz 2024-09-14T23:27:46.899 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43123.log 2024-09-14T23:27:46.900 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66029.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66029.log.gz 2024-09-14T23:27:46.900 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.36023.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36023.log.gz 2024-09-14T23:27:46.900 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59614.log 2024-09-14T23:27:46.906 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.43123.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43123.log.gz 2024-09-14T23:27:46.907 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40834.log 2024-09-14T23:27:46.907 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.59614.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59614.log.gz 2024-09-14T23:27:46.908 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51730.log 2024-09-14T23:27:46.909 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.40834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.39819.log 2024-09-14T23:27:46.911 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51730.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51730.log.gz 2024-09-14T23:27:46.911 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44436.log 2024-09-14T23:27:46.912 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.3.39819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68249.log 2024-09-14T23:27:46.918 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44436.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44436.log.gz 2024-09-14T23:27:46.920 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50129.log 2024-09-14T23:27:46.921 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68249.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68249.log.gz 2024-09-14T23:27:46.921 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48179.log 2024-09-14T23:27:46.922 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48634.log 2024-09-14T23:27:46.930 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48179.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48179.log.gz 2024-09-14T23:27:46.931 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73007.log 2024-09-14T23:27:46.931 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48634.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48634.log.gz 2024-09-14T23:27:46.935 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50487.log 2024-09-14T23:27:46.936 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.73007.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73007.log.gz 2024-09-14T23:27:46.936 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53968.log 2024-09-14T23:27:46.937 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50487.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50487.log.gz 2024-09-14T23:27:46.940 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57653.log 2024-09-14T23:27:46.940 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.53968.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53968.log.gz 2024-09-14T23:27:46.942 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57653.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57653.log.gz 2024-09-14T23:27:46.944 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47452.log.gz 2024-09-14T23:27:46.944 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70257.log 2024-09-14T23:27:46.944 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44459.log 2024-09-14T23:27:46.944 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70487.log 2024-09-14T23:27:46.944 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70257.log.gz/var/log/ceph/cluster2-client.mirror.44459.log: 2024-09-14T23:27:46.945 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70487.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70487.log.gz 2024-09-14T23:27:46.948 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42443.log 2024-09-14T23:27:46.948 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48380.log 2024-09-14T23:27:46.950 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42443.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44459.log.gz 2024-09-14T23:27:46.952 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55212.log 2024-09-14T23:27:46.953 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42443.log.gz 2024-09-14T23:27:46.957 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39459.log 2024-09-14T23:27:46.957 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.55212.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55212.log.gz 2024-09-14T23:27:46.965 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65509.log 2024-09-14T23:27:46.970 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.39459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42053.log 2024-09-14T23:27:46.974 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65509.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65509.log.gz 2024-09-14T23:27:46.983 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61668.log 2024-09-14T23:27:46.987 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35917.log 2024-09-14T23:27:46.988 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.61668.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61668.log.gz 2024-09-14T23:27:46.988 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38751.log 2024-09-14T23:27:46.991 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35917.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35917.log.gz 2024-09-14T23:27:46.992 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39576.log 2024-09-14T23:27:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.38751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57787.log 2024-09-14T23:27:46.997 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74504.log 2024-09-14T23:27:47.002 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57787.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46418.log.gz 2024-09-14T23:27:47.002 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57787.log.gz 2024-09-14T23:27:47.004 INFO:teuthology.orchestra.run.smithi089.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38751.log.gz 2024-09-14T23:27:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39819.log 2024-09-14T23:27:47.007 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50374.log 2024-09-14T23:27:47.007 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.39819.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74504.log.gz 2024-09-14T23:27:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74484.log 2024-09-14T23:27:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50374.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50374.log.gz 2024-09-14T23:27:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66049.log 2024-09-14T23:27:47.011 INFO:teuthology.orchestra.run.smithi089.stderr: 73.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39819.log.gz 2024-09-14T23:27:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64144.log 2024-09-14T23:27:47.011 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74484.log.gz 2024-09-14T23:27:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67979.log 2024-09-14T23:27:47.012 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.64144.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64144.log.gz 2024-09-14T23:27:47.015 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74525.log 2024-09-14T23:27:47.015 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67979.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67979.log.gz 2024-09-14T23:27:47.015 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56591.log 2024-09-14T23:27:47.015 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.74525.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66049.log.gz 2024-09-14T23:27:47.015 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74525.log.gz 2024-09-14T23:27:47.016 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49098.log 2024-09-14T23:27:47.016 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.56591.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56591.log.gz 2024-09-14T23:27:47.019 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42301.log 2024-09-14T23:27:47.019 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65365.log 2024-09-14T23:27:47.020 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.42301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66653.log 2024-09-14T23:27:47.020 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65365.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65365.log.gz 2024-09-14T23:27:47.023 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58755.log 2024-09-14T23:27:47.024 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.66653.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66653.log.gz 2024-09-14T23:27:47.032 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57215.log 2024-09-14T23:27:47.032 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.58755.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58755.log.gz 2024-09-14T23:27:47.032 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47190.log 2024-09-14T23:27:47.036 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57215.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57215.log.gz 2024-09-14T23:27:47.041 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59775.log 2024-09-14T23:27:47.041 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47190.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47190.log.gz 2024-09-14T23:27:47.041 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65737.log 2024-09-14T23:27:47.045 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59775.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59775.log.gz 2024-09-14T23:27:47.050 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63876.log 2024-09-14T23:27:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65737.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65737.log.gz 2024-09-14T23:27:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50511.log 2024-09-14T23:27:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63876.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63876.log.gz 2024-09-14T23:27:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61022.log 2024-09-14T23:27:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.50511.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50511.log.gz 2024-09-14T23:27:47.063 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46782.log 2024-09-14T23:27:47.072 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52079.log 2024-09-14T23:27:47.073 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.46782.log: 87.4% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45653.log.gz 2024-09-14T23:27:47.080 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61022.log.gz 2024-09-14T23:27:47.080 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46782.log.gz 2024-09-14T23:27:47.080 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65116.log 2024-09-14T23:27:47.081 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69233.log 2024-09-14T23:27:47.081 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65116.log: /var/log/ceph/cluster1-client.mirror.52079.log: 87.4% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65116.log.gz 2024-09-14T23:27:47.081 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52079.log.gz 2024-09-14T23:27:47.081 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49341.log 2024-09-14T23:27:47.081 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.69233.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69233.log.gz 2024-09-14T23:27:47.081 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47371.log 2024-09-14T23:27:47.081 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.49341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61502.log 2024-09-14T23:27:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.47371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57450.log 2024-09-14T23:27:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.61502.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61502.log.gz 2024-09-14T23:27:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45123.log 2024-09-14T23:27:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57450.log: 88.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68182.log 2024-09-14T23:27:47.108 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57450.log.gz 2024-09-14T23:27:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.54046.log 2024-09-14T23:27:47.121 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45123.log.gz 2024-09-14T23:27:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.68182.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68182.log.gz 2024-09-14T23:27:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54340.log 2024-09-14T23:27:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.3.54046.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48930.log.gz 2024-09-14T23:27:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57239.log 2024-09-14T23:27:47.147 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54340.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54340.log.gz 2024-09-14T23:27:47.165 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50332.log 2024-09-14T23:27:47.166 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.57239.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57239.log.gz 2024-09-14T23:27:47.183 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57964.log 2024-09-14T23:27:47.183 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.50332.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50332.log.gz 2024-09-14T23:27:47.196 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48693.log 2024-09-14T23:27:47.197 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.57964.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57964.log.gz 2024-09-14T23:27:47.202 INFO:teuthology.orchestra.run.smithi089.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.54046.log.gz 2024-09-14T23:27:47.206 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54938.log 2024-09-14T23:27:47.216 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51424.log 2024-09-14T23:27:47.217 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.54938.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54938.log.gz 2024-09-14T23:27:47.230 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36465.log 2024-09-14T23:27:47.231 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.51424.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51424.log.gz 2024-09-14T23:27:47.237 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65157.log 2024-09-14T23:27:47.246 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.admin.36465.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48952.log 2024-09-14T23:27:47.250 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.65157.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36465.log.gz 2024-09-14T23:27:47.255 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35916.log 2024-09-14T23:27:47.259 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.48952.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65157.log.gz 2024-09-14T23:27:47.261 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49098.log.gz 2024-09-14T23:27:47.263 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42511.log 2024-09-14T23:27:47.263 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.admin.35916.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35916.log.gz 2024-09-14T23:27:47.263 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67777.log 2024-09-14T23:27:47.263 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.42511.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42511.log.gz 2024-09-14T23:27:47.265 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44307.log 2024-09-14T23:27:47.265 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67777.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67777.log.gz 2024-09-14T23:27:47.265 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63523.log 2024-09-14T23:27:47.268 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44307.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48952.log.gz 2024-09-14T23:27:47.270 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44307.log.gz 2024-09-14T23:27:47.270 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67439.log 2024-09-14T23:27:47.272 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.63523.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63523.log.gz 2024-09-14T23:27:47.272 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70378.log 2024-09-14T23:27:47.273 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.67439.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67439.log.gz 2024-09-14T23:27:47.274 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52348.log 2024-09-14T23:27:47.274 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.70378.log: 87.8% 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41578.log.gz 2024-09-14T23:27:47.274 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70378.log.gz 2024-09-14T23:27:47.276 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44522.log 2024-09-14T23:27:47.276 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.52348.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52348.log.gz 2024-09-14T23:27:47.276 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37787.log 2024-09-14T23:27:47.282 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.44522.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44522.log.gz 2024-09-14T23:27:47.282 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45344.log 2024-09-14T23:27:47.283 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.37787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59092.log 2024-09-14T23:27:47.284 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45344.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45344.log.gz 2024-09-14T23:27:47.284 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45023.log 2024-09-14T23:27:47.285 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.59092.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48380.log.gz 2024-09-14T23:27:47.286 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61367.log 2024-09-14T23:27:47.288 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.45023.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45023.log.gz 2024-09-14T23:27:47.288 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69689.log/var/log/ceph/cluster1-client.mirror.61367.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37787.log.gz 2024-09-14T23:27:47.288 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:27:47.288 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40834.log.gz 2024-09-14T23:27:47.288 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61367.log.gz 2024-09-14T23:27:47.290 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41264.log.gz 2024-09-14T23:27:47.300 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48693.log.gz 2024-09-14T23:27:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72214.log 2024-09-14T23:27:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.69689.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69689.log.gz 2024-09-14T23:27:47.305 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60207.log 2024-09-14T23:27:47.305 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52232.log 2024-09-14T23:27:47.305 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.60207.log: /var/log/ceph/cluster2-client.mirror.72214.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72214.log.gz 2024-09-14T23:27:47.306 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62381.log 2024-09-14T23:27:47.306 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60207.log.gz 2024-09-14T23:27:47.306 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41156.log 2024-09-14T23:27:47.306 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster2-client.mirror.52232.log: /var/log/ceph/cluster1-client.mirror.62381.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62381.log.gz 2024-09-14T23:27:47.306 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52232.log.gz 2024-09-14T23:27:47.307 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cluster1-client.mirror.41156.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.39459.log.gz 2024-09-14T23:27:47.309 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42053.log.gz 88.3% 2024-09-14T23:27:47.309 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41156.log.gz 2024-09-14T23:27:47.309 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59092.log.gz 2024-09-14T23:27:47.328 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39146.log.gz 2024-09-14T23:27:47.330 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46579.log.gz 2024-09-14T23:27:47.418 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41715.log.gz 2024-09-14T23:27:47.429 INFO:teuthology.orchestra.run.smithi089.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39576.log.gz 2024-09-14T23:27:47.460 INFO:teuthology.orchestra.run.smithi089.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.39819.log.gz 2024-09-14T23:27:47.469 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71905.log.gz 2024-09-14T23:27:47.472 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50129.log.gz 2024-09-14T23:27:47.481 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47371.log.gz 2024-09-14T23:27:47.485 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49341.log.gz 2024-09-14T23:27:47.672 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41201.log.gz 2024-09-14T23:27:48.824 INFO:teuthology.orchestra.run.smithi089.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36856.log.gz 2024-09-14T23:27:48.960 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38485.log.gz 2024-09-14T23:27:49.067 INFO:teuthology.orchestra.run.smithi089.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-14T23:27:49.174 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39683.log.gz 2024-09-14T23:27:50.245 INFO:teuthology.orchestra.run.smithi052.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-14T23:27:50.381 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42301.log.gz 2024-09-14T23:27:50.582 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41802.log.gz 2024-09-14T23:27:51.397 INFO:teuthology.orchestra.run.smithi052.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-14T23:27:52.055 INFO:teuthology.orchestra.run.smithi052.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-14T23:27:52.057 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-14T23:27:52.057 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m9.164s 2024-09-14T23:27:52.057 INFO:teuthology.orchestra.run.smithi052.stderr:user 0m25.695s 2024-09-14T23:27:52.057 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m0.892s 2024-09-14T23:27:54.506 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40194.log.gz 2024-09-14T23:29:23.639 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-14T23:29:30.890 INFO:teuthology.orchestra.run.smithi089.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-14T23:29:49.635 INFO:teuthology.orchestra.run.smithi089.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-14T23:29:49.637 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-14T23:29:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:real 2m6.711s 2024-09-14T23:29:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:user 3m53.691s 2024-09-14T23:29:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m10.255s 2024-09-14T23:29:49.637 INFO:tasks.ceph:Archiving logs... 2024-09-14T23:29:49.713 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733/remote/smithi052/log 2024-09-14T23:29:49.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T23:29:53.221 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733/remote/smithi089/log 2024-09-14T23:29:53.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T23:30:27.998 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-14T23:30:28.009 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-14T23:30:28.009 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T23:30:28.012 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T23:30:28.106 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-14T23:30:28.117 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-14T23:30:28.117 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T23:30:28.119 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T23:30:28.134 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-09-14T23:30:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-09-14T23:30:28.139 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T23:30:28.139 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-09-14T23:30:28.139 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +879us[ +882us] +/- 64ms 2024-09-14T23:30:28.139 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 +921us[ +924us] +/- 63ms 2024-09-14T23:30:28.139 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 -2005us[-2002us] +/- 69ms 2024-09-14T23:30:28.139 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T23:30:28.175 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T23:30:28.175 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-09-14T23:30:28.175 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +865us[ +862us] +/- 64ms 2024-09-14T23:30:28.175 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 +905us[ +902us] +/- 63ms 2024-09-14T23:30:28.175 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -2002us[-2004us] +/- 69ms 2024-09-14T23:30:28.175 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T23:30:28.176 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-14T23:30:28.186 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-14T23:30:28.187 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-14T23:30:28.198 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-14T23:30:28.268 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-14T23:30:28.389 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-14T23:30:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T23:30:28.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-14T23:30:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T23:30:28.450 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-14T23:30:28.461 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-14T23:30:28.471 INFO:teuthology.task.internal:Duration was 2887.255364 seconds 2024-09-14T23:30:28.472 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-14T23:30:28.483 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-14T23:30:28.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T23:30:28.486 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T23:30:28.523 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T23:30:28.555 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T23:30:28.670 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-14T23:30:28.671 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-09-14T23:30:28.671 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T23:30:28.732 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-09-14T23:30:28.732 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T23:30:28.753 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-14T23:30:28.753 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T23:30:28.774 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T23:30:28.819 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-14T23:30:28.819 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T23:30:28.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T23:30:28.970 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-14T23:30:28.983 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-14T23:30:28.983 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T23:30:29.010 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T23:30:29.043 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-14T23:30:29.056 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T23:30:29.059 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T23:30:29.081 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-09-14T23:30:29.119 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-09-14T23:30:29.239 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T23:30:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T23:30:29.295 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T23:30:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T23:30:29.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-14T23:30:29.322 INFO:teuthology.task.internal:Transferring archived files... 2024-09-14T23:30:29.323 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733/remote/smithi052 2024-09-14T23:30:29.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T23:30:29.369 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-14_21:16:02-rbd-squid-distro-default-smithi/7904733/remote/smithi089 2024-09-14T23:30:29.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T23:30:29.402 INFO:teuthology.task.internal:Removing archive directory... 2024-09-14T23:30:29.402 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T23:30:29.406 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T23:30:29.452 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-14T23:30:29.464 INFO:teuthology.task.internal:Not uploading archives. 2024-09-14T23:30:29.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-14T23:30:29.500 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-14T23:30:29.501 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T23:30:29.503 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T23:30:29.516 INFO:teuthology.orchestra.run.smithi052.stdout: 265895 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 23:30 /home/ubuntu/cephtest 2024-09-14T23:30:29.517 INFO:teuthology.orchestra.run.smithi089.stdout: 265325 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 23:30 /home/ubuntu/cephtest 2024-09-14T23:30:29.518 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-14T23:30:29.531 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-14T23:30:29.557 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} duration: 2887.255363702774 failure_reason: '"2024-09-14T22:51:40.227270+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-09-14T23:30:29.557 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T23:30:29.637 INFO:teuthology.run:FAIL